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(57) ABSTRACT 

An independent checkpointing method using a memory 
checkpoint on a distributed system that includes a message 
transmission routine, a message processing routine, and a 
periodical checkpoint routine. The message transmission 
routine adds a self checkpoint number to a message to be 
transmitted when a current process tries to send a message 
to another process. The message processing routine per- 
forms a memory checkpoint and processes a message in 
reference to a checkpoint number of a transmission process, 
a checkpoint number of the current process, a memory 
checkpoint flag, and a message transmission flag when a 
message is received from a process. The periodical check- 
point routine performs a checkpoint that records a necessary 
state information for recovery against faults periodically in 
reference to the memory checkpoint flag. 

4 Claims, 4 Drawing Sheets 
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INDEPENDENT CHECKPOINTING METHOD and a periodical checkpoint routine. The message transmis- 

USING A MEMORY CHECKPOINT ON A sion routine adds a self checkpoint number to a message to 

DISTRIBUTED SYSTEM transmitted when a current process tries to send a 

message to another process. The message processing routine 
5 performs a memory checlqjoint and processes a message io 

TECHNICAL FIELD reference to a checkpoint number of a transmission process, 

a checkpoint number of the current process, a memory 

The present invention relates to an independent check- checkpoint flag, and a message transmission flag when a 
pointing method, which recovers problems in running a message is received from a process. The periodical check- 
distributed system and reduces work time. In the method in point routine performs a checkpoint that records a necessary 
accordance with an embodiment of the present invention, ^xsl\& information for recovery against faults periodically in 
transmitting processes send messages with a self checlqjoint reference to the memory checkpoint flag, 
number, and receiving processes determine whether Preferably, the message transmission routine includes the 
memory checkpomt is to be performed m reference to the following steps. A step is to generate a message to be 
transmittmg processes checkpomt number, current process transmitted. Another step is to add the checkpoint number of 
checkpomt number memory checkpomt flag, and message ^^^^^t process to the message to be transmitted. A 
traiKmission flag before proccssmg the received message In ^^^^^^ transmission flag true for 
performmg penodical cbeckpomtmgs. the method inakcs preparing cases in which an orphan message occurs. An 
reference to the result of the memory checkpomt for check- additional step is to transmit the message, 
pomtmg. Preferably, the message processing routine includes the 
BACKGROUND OF THE INVENTION following steps. A first step is to receive the message from 

the process and compare the checkpoint nimaber of the 

Checkpointing technology stores status information of transmission process with the checkpoint number of the 
each process and recovers errors by using the stored infor- current process. Another step is to process the received 
mation when errors occur in running distributed system 25 message if the checkpoint number of the transmission pro- 
environments, cess is smaller than or equal to the checkpoint number of the 

Several studies have been done in the area of checkpoint- current process. A further step is to check the memory 

ing technology. An adaptive checkpointing algorithm was checkpoint flag if the checkpoint number of the transmission 

proposed by Jian Xu et al. at Proceedings of Fifth IEEE process is larger than the checkpoint number of the current 

Symposium on Parallel and Distributed Processing in 1993 . 30 process. Another step is to replace the checkpoint number of 

The adaptive checkpointing algorithm performs checkpoint- the current process with the checkpoint number of the 

ing by checking whether zigzag cycles made by an input message transmission process and process the received 

messa^ exist. If input messages make zigzag cycles, the message if the checked memory chec]q>oint flag is true, A 

adaptive checkpointing algorithm is based upon the fact that further step is to check the message transmission flag if the 

a domino effect may be caused. Therefore, if input messages 35 checked memory checkpoint flag is false. An additional step 

make zigzag cycles, the adaptive checkpointing algorithm is to record the state information of the current process into 

performs checlqpointing to remove zigzag cycles before the a memory, set the checkpoint number of the current process 

processing of messages. as the checkpoint number of message transmission process, 

A lazy checkpointing algorithm was proposed by Wang et ^et the memory checkpoint flag true, set the message irans- 

al. at Technical report CRHC-92-27. In the lazy checkpoint- ^ "fission flag false, and process the received message if the 

ing algorithm, message transmission processes attach a self checked message transmission flag is true. Another addi- 

checkpoiniing number to messages to be transmitted, and ^lep is to replace the checkpoint number of the current 

the message receiving processes compare the checkpointing process with the checkpoint number of the transmission 

number of the transmitted messages with their checkpoint process and process the received if the checked message 

number before processing the transmitted messages. If the ^5 transmission flag is false. 

checkpoint number of the transmission process is larger, Preferably, the periodical checkpoint routine includes the 

checkpointing is performed before the message is processed. following steps. An initial step is to check the memory 

At the moment, the checkpoint numbers of the two processes checkpoint flag on a periodical checkpoint time. Another 

become identical. ^tep is to record the state information stored at the memory 

However, when the amount of message transmission and ^ ^ '{^^^ "'^["^^ checkpoint flag is true or 

execution speed difference increases, these checkpoinUng recordmg the state informatton to a disk and increasmg the 

methods may cause a large number of checkpoints and checkix)mt number by one if the checked memory check- 

increase job completion time. pomt flag is false. A further step is to calculate a next 

_ . rL,. ... , periodical checkpomt time. An additional step is to set the 

The number of checkpomts is directly related with job r«or«™ ti,» ™-c.c-„„^ *«iL™;c-^™ fl«„ 

. , . .. t. memory checkpomt nag and the message transmission flag 
completion time in the error-free environment and the roU 

back distance is directly related with job completion lime in 

the environment with errors. Therefore, a large number of BRIEF DESCRIPTION OF THE DRAWINGS 

checkpoints and an increase in roll back distance causes The embodiments of the present invention will be 

delayed job completion time. ^ explained with reference to the accompanying drawings, in 

which: 

FIG. 1 is a diagram illustrating a process in a distributed 

An independent checkpointing method using memory system in accordance with an embodiment of the present 

checkpoint on a distributed system is provided. invention; 

The independent checkpointing method in accordance 65 FIG. 2 is a flow diagram illustrating a message transmis- 

with an embodiment of the present invention includes a sion routine in accordance with an embodiment of the 

message transmission routine, a message processing routine, present invention; 



SUMMARY OF THE INVENTION 
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FIG. 3 is a flow diagram illustrating a message processing 
routine in accordance with an embodiment of ibe present 
invention; and 

FIG. 4 is a flow diagram illustrating a periodical check- 
pointing routine in accordance with an embodiment of the ^ 
present invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

The method in accordance with an embodiment of the 
present invention may be applied to a distributed system, 
which includes a number of local machines connected 
through LANs (local area network). 

FIG. 1 is a diagram illustrating a process in a distributed 15 
system in accordance with an embodiment of the present 
invention. As shown in FIG. 1, a local machine includes 
local memory and a local disk. A local machine executes a 
process allocated to a job. Each process includes message 
transmission routine, message processing routine, and peri- 20 
odical checkpoint routine for recovery against faults. Each 
process sends information to other processes through mes- 
sages. 

FIG. 2 is a flow diagram illustrating a message transmis- 
sion routine in accordance with an embodiment of the 25 
present invention. As shown in FIG. 2, the independent 
checkpointing method includes four steps. A flrst step is to 
generate a message to be transmitted. A second step is to 
attach checkpoint number of current process to the message. 
A third step is to set transmission flag true for processing 30 
orphan messages. A fourth step is to transmit the message. 

When the current process is running and a new message 
needs to be transmitted to a different process running in a 
different local machine, a message that needs to be trans- 
mitted earlier is generated at step SIO. 

Then, a checkpoint number of the current process is added 
to the generated message to be transmitted at step Sll. 

Executing a periodical checkpoint routine that is to be 
described later may increase the checkpoint number of the ^ 
current process. When a message arrives from other 
processes, the checkpoint number of the current process may 
be replaced by the checkpoint number of the transmission 
process. 

After the checkpoint number is added to the message to be ^5 
transmitted, the message transmission flag is set true for 
notifying that the current process has sent a message to 
another process and then the message is transmitted at step 
S12 and S13. 

The message transmission flag is important because it is 50 
necessary to reduce the number of the memory checkpoint 
when orphan messages are generated. Orphan messages are 
the messages transmitted after the checkpoint of the message 
transmission processes and received before the checkpoint 
of the message receiving processes. When errors occur in the 55 
message transmission processes, orphan messages bring 
back the message receiving processes to the point before the 
checlqjoints. Therefore, in order to avoid generating orphan 
messages, checkpointing is performed after the message 
transmission processes send a message. go 

The method in accordance with an embodiment of the 
present invention employs message transmission flags to 
reduce the number of memory checlqjoints in overall execu- 
tion of each process and to prevent orphan messages. That 
is, memory checkpoint is performed only if the checkpoint 65 
number of the message transmission process is larger than 
checkpoint number of the message receiving process and the 
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current message transmission flag of the message receiving 
process is true. 

FIG. 3 is a flow diagram illustrating a message processing 
routine in accordance with an embodiment of ibc present 
invention. 

As shown in FIG. 3, the message processing routine 
includes the following seven steps. A first step is to receive 
a message from a process and compare a checkpoint number 
of the message transmission process with a checkpoint 
number of the current process. A second step is to process 
the received message if the checkpoint number of the 
message transmission process is smaller than or equal to the 
checkpoint number of the current process. A third step is to 
check a memory checkpoint flag if the checkpoint number of 
the message transmission process is larger than the check- 
point number of the current process. A fourth step is to 
replace the checkpoint number of the current process with 
the checkpoint number of the message transmission process 
and process the received message if the memory checkpoint 
flag checked is true. A fifth step is to check the message 
transmission flag if the memory checkpoint flag checked is 
false, A sixth step is to record state information of the current 
process into memory, set the checlqwint number of the 
current process as a checkpoint number of message trans- 
mission process, set the memory checkpoint flag true, set the 
message transmission flag false, and process the received 
message if the message transmission flag checked is true. A 
seventh step is to replace the checkpoint number of the 
current process with a checkpoint number of the message 
transmission process and process the received message if the 
message transmission flag checked is false. 

When a message is received from other processes, the 
message processing routine compares the checkpoint num- 
ber of the current process with the checkpoint number of the 
message transmission process that is added to the message 
before processing the received message at step S21 and S22. 

If the checkpoint number of the message transmission 
process is smaller than or equal to the checkpoint number of 
the current process, it means that the processing speed of the 
message transmission process is slower than or equal to the 
processing speed of the current process. In such cases, since 
the memory checkpoint of the current process is not 
necessary, the current process treats the received message 
without performing the memory checkpoint at step S25. 

However, if the checkpoint number of the message trans- 
mission process is larger than the checkpoint number of the 
current process, it means that the processing speed of the 
message transmission process is faster than the processing 
speed of the current process. That is, the current process 
determines whether the memory checkpoint is to be per- 
formed before processing the received message. 

The memory checkpoint is performed only if the check- 
point number of the message transmission process is larger 
than the checkpoint number of the current process, the 
memory checkpoint has not been performed, and the 
memory checkpoint after transmitting a message to other 
processes and periodical checkpoint have not been per- 
formed. It reduces number of checkpoints by preventing 
repetition of memory chedcpoints and therefore reduces job 
completion time, even if a number of messages are received 
from other processes before the periodical checlq5oint is 
performed. 

For this purpose, the memory checkpoint flag is checked 
to see whether the memory checkpoint has been performed 
before at step S23. 

If the flag is true, which means the memory checkpoint 
was performed after the periodical checlqjoint is performed, 
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the memory checkpoint is not to be perfiDrmed. Instead, the 
checkpoint number of the current process is replaced by a 
checkpoint number of the message transmission process and 
then a next message is processed at steps S24 and S25. 

If the Qag is false at step S23, the message transmission 
flag is confinned at step S26. 

The message transmission flag is a flag describing if the 
current process has sent any messages to other processes. 
The message transmission flag is set as true by the message 
transmission routine and as false after the memory check- 
point or periodical checkpoint is performed. 

Therefore, if the message transmissbn flag is true, which 
means that the result of the step S26 is true, that is, memory 
checkpoint or periodical checkpoint is not performed yet, 
the memory checkpoint that stores the state information of 
the current process into the memory is performed. Check- 
point numbers become identical by replacing checkpoint 
number by checkpoint number of message transmission 
process at step S27. 

Then, the memory checkpoint flag is set true and the 
message transmission flag value is set false at steps S28 and 
S25 before the next message is processed. 

HG. 4 is a flow diagram illustrating a periodical check- 
pointing routine in accordance with an embodiment of the 
present invention. As shown in FIG. 4, the periodical check- 
pointing routine includes the following steps. A first step is 
to check the memory checlqjoint flag on periodical check- 
point time. A second step is to record process state infor- 
mation stored at memory to disk if the memory checkpoint 
flag checked is true or recording process state information to 
disk and ino-easing the checkpoint number by one if the 
memory chedcpoint flag checked is false. A third step is to 
calculate a next periodical checkpoint time. A fourth step is 
to set the memory checkpoint flag and the message trans- 
mission flag false. 

The periodical checkpoint routine checks whether the 
memory checkpoint flag is true or false before writing 
current state information of the process with a certain period 
at steps S31 and S32. 

The memory checkpoint flag is a flag indicating the 
memory checkpoint has been performed in the message 
processing routine. If the memory checkpoint flag is true, 
which means the memory checkpoint has been performed 
recently, the process state information stored in memory is 
written to the disk without increasing the checkpoint nimiber 
at step S32. 

If the memory checkpoint flag is false, which means the 
memory checkpoint has not been performed recently, the 
process state information is written on the disk while 
increasing the checkpoint number by 1 at step S33. 

Then, the amount of time for executing the periodical 
checkpoint is calculated and the memory checkpoint flag 
and message transmission flag arc set as false at steps S34 
through S36. 

When the amount of message transmission and execution 
speed difference increase, conventional checlq)ointing meth- 
ods cause a large number of checkpoints, which increases 
delay caused by checkpoints and eventually causes 
increased job completion time. 

However, the method in accordance with an embodiment 
of the present invention only performs checkpoint on the 
memory when a message is initially received &Dm a process 
with a larger checkpoint number. Later, the method increases 
only the checkpoint number without performing any check- 
points. As shown in FIG. 4, process state information stored 
in the memory is recorded to disks in periodical checlqwints. 
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That is, process state information is recorded to disks with 
a certain period. Even though memory dieckpoint is caused 
in every period, the time for memory checlq)oini is substan- 
tially smaller than the time for the disk checkpoint, which 
5 means that actual job delay time is trivial. 

On the contrary, to the conventional checkpoint methods, 
the method in accordance with an embodiment of the present 
invention performs only one disk checkpoint with a certain 
period. 

^° In addition, as shown in FIG. 3, the method in accordance 
with an embodiment of the present invention removes 
dependent relations between processes that are usually 
caused by message transmissions by comparing checkpoint 
numbers. That is, when a process receives a message with a 
larger checkpoint number than the process, the process 
performs the memory checkpoint before processing the 
message and identifies a checkpoint number with the trans- 
mission process. This procedure virtually removes differ- 
ence of execution speeds and reduces dependent relations 

2^ between processes. As a result, the method in accordance 
with an embodiment of the present invention has a short roll 
back distance when errors occur. 

As described above, the method of the present invention 
adds a checkpoint number in transmitting messages between 
processes. When a chedcpoint number of message transmis- 
sion process is larger than a checkpoint number of a current 
process, the method of the present invention performs the 
memory checkpoint in reference to a memory checkpoint 
flag and message transmission flag, replaces the checkpoint 
number of the current process with a checkpoint number of 
the message transmission process. In addition, the method of 
the present invention uses the result of the memory check- 
point in performing a periodical checkpoint. 

2j Therefore, when a number of processes perform the same 
jobs in a distributed system environment, the method in 
accordance of an embodiment of the present invention has 
minimum nimiber of checkpoints regardless of the amount 
of message transmission and execution speed difference. 

^ Also, the method removes dependent relations between 
processes that are usually caused by message transmissions 
by comparing checkpoint numbers. The method in accor- 
dance with an embodiment of the present invention has a 
short roll back distance when errors occur. Consequently, the 
method is effective for completing jobs in a minimum 
amount of time in a distributed system environment with 
errors or in a distributed system environment without errors. 

Although representative embodiments of the present 
invention have been disclosed for illustrative purpose, those 

50 who arc skilled in the art will appreciate that various 
modifications, additions and substitutions are possible with- 
out departing from the scope and spirit of the present 
invention as defined in the accompanying claims and in 
equivalents thereof. 

55 What we claim: 

1. An independent checkpointing method using a memory 
checkpoint on a distributed system, comprising: 

a message transmission routine for adding a self check- 
point number to a message to be transmitted when a 

60 current process tries to send a message to another 
process; 

a message processing routine for performing a memory 
checkpoint and processing a message in reference to a 
checlq)oint number of a transmission process, a check- 
65 point number of the current process, a memory check- 
point flag, and a message transmission flag when a 
message is received from a process; and 
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a periodical checkpoint routine for performing a check- 
point that records a necessary state information for 
recovery against faults periodically in reference to the 
memory checlq)oint flag. 

2, The independent checkpointing method of claim 1, 5 
udierein the message transmission routine comprises the 
steps of: 

generating the message to be transmitted; 
adding the checkpoint number of the current process to 
the message; 

setting the message transmission flag to true for preparing 

cases in which an orphan message occurs; and 
transmitting the message. 

3. The independent checkpointing method of claim 1, 15 
wherein the message processing routine comprises the steps 
of: 

receiving the message from the process and comparing 
the checkpoint number of the transmission process with 
the checlqjoint number of the current process; 20 

processing the received message if the checkpoint number 
of the transmission process is smaller than or equal to 
the checkpoint number of the current process; 

checking the memory checkpoint flag if the checkpoint 
number of the transmission process is larger than the 
checkpoint number of the current process; 

replacing the checkpoint number of the current process 
with the checkpoint number of the message transmis- 
sion process and processing the received message if the 
checked memory checkpoint flag is true; 
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checking the message transmission flag if the checked 
memory checkpoint flag is false; 

recording the stale information of the current process into 
a memory, setting the checkpoint number of the current 
process as the checkpoint number of message trans- 
mission process, setting the memory checkpoint flag 
true, setting the message transmission flag false, and 
processing the received message if the checked mes- 
sage Uansmission flag is true; and 

replacing the checkpoint number of the current process 
with the checkpoint number of the message transmis- 
sion process and processing the received message if the 
checked message transmission flag is false. 

4. The independent checkpointing method of claim 3, 
wherein the periodical checkpoint routine comprises the 
steps of: 

checking the memory checlq)oint flag on a periodical 
checkpoint time; 

recording the state information stored in the memory to a 
disk if the checked memory checkpoint flag is true or 
recording the state information to a disk and increasing 
the checkpoint number by one if the checked memory 
checkpoint flag is false; 

calculating a next periodical checkpoint time; and 

setting the memory checkpoint flag and the message 
transmission flag false. 

« « * * « 
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ABSTRACT 



A method of managing, in a telecommunication net- 
work, user data of a user who may move from a base 
work area associated with a base exchange which com- 
prises a base memory to another work area associated 
with another exchange which comprises another mem- 
ory, independent of whether or not the user is commu- 
nicating. The method con^irises effecting a coupling 
between the base memory and the another memory 
when the user moves from the base work area to the 
another work area; copying a largest portion of the user 
data stored in the base memory and storing the copied 
data in the another memory via the coupling, the user 
data stored in the base memory remaining avaflable for 
further copying, and changes to be made in the user 
data taking place both in the base memory and in the 
another memory. The copied portion of the user data 
stored in the another memory is inspected for correct- 
ness v^th respect to the user data stored in the base 
memory and, in the event of correctness of the copied 
portion of the user data, the portion of the user data 
stored in the another memory is made available for 
further copying. Oianges to be made in the data take 
place only in the another memory, and the copied por- 
tion of-ihe user data stored in the base memory thereaf- 
ter becomes unavailable for further copying. 

16 Claims, 1 Drawing Sheet 
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ory, a largest portion of the user data stored in the base 

METHOD OF MANA GING , IN A ' memory being copied and stored via the coupling in the 

raLECOMMUNICATION NETWORK. USER other memory and the user data stored in the base mem- 

DATA OF A USER WHO MAY MOVE FROM A ory being definitive Ci e,, being available for copying) 

BASE WORK AREA ASSOCIATED Wrra A BASE 5 for the user and. changes to be made in the user data 

EXCHANGE TO ANOTHER WORK AREA taking place both in the base memory and in the other 

ASSOCIATED WITH ANOTHER EXCHANGE memory, the portion of the user data copied into the 

WAr'vi-ur^TTTsm nr? xm? rtavTSNrrrriM other memory then being mspected for correctness with 

BACKGROUND OF THE INVENTION respect to the user data stored m the base memory. 

The invention rdates to a method of managing, in a whereafter, in the event of correctness, the portion of 

telecommunication network, user data of a user who the user data stored in the other memory becomes defin- 

may move &om a base work area associated with a base itive for the user ^.e., becomes available for further 

exchange which comprises base memory means to an- copying) and changes to be made in the data take place 

other work area associated with another exchange only in the other memory means, the copied portion of 

which comprises other memory means. ' 5 ^ stored in the base memory becoming negh- 

A method of this type is disclosed in various GSM gible 0-e., becoming unavailable for further copying). 
(Group Sp6dale Mobile) recommendations, such as The method according to the invention rejects the 
GSM 03.03, 03.08, 11.30, 11.31 and 11.32. According to constant fimctionmg of the base memory as "master" 
the method described thercm, the base monory means memory and the constant functionmg of the other mem- 
associated with the base exchange (Home Location 20 ^^y as "slave" memory means as regards a particular 
Register or HLR) comprise all the data of the user who, user. Because the largest portion of the user data stored 
normaUy spe^g, is located in the base work area ^ the base memory is copied and stored in the other 
associated with the base exchange. K the u^^ moves to ^lemory. the copied portion of the user data stored in 
the other workareaassocmted with t^ the other memory is then inspected for correctness with 
only those data which are required for handhng caUs 25 ^ ^ ^^^^^ ^ ^ 

andforotherpiuposesCwtehareof^^^^ portion of the user data stored in the other m«^ory 
user tempormly located m said other work area) are ^^^^ definitive (available for copying) for the 
copied out of the base memory means (HLR) mto the i,** *i « 
other memory means associated with the other ex- correctaess has been confirmed and the copied 
change (Visitor Location Register or VLR). Under 30 °° ^^^^^ ^ ^^red m the base memory 
thesTcircumstances, the base memory means (HLR) ^^^^ negligible and consequently ov^- 
function as "master*' memory means and the other y^^eable, the memory capacity required (unavailable 
memory means (VLR) function as "slave" memory ^""^ wpymg) for said user dato m the base memory is 
means, some user data thus bemg present in two differ- saved, which mcreases the efficiency of the method 
ent memories while the user is located m the work area 35 according to the mvention. If the user then moves to a 
associated with the other exchange.. If the user moves to succeedmg work area associated with a succeeding 
a further work area associated with a further exchange, exchange which comprises a succeedmg memory, the 
agam only those data which are required for handling largest portion of the user data stored m the other mem- 
calls and for other purposes (which are of importance copied and stored in the succeeding memory. The 
for the user temporarily located in the said further work 40 copied portion of the user data stored in the succeeding 
area) are copied fi-om the base memory means (HLR) memory is subsequently inspected for correctness with 
into the further memory means (VLR) associated with respect to the user data stored in the other memory, the 
the further exchange, the base memory means (HLR) portion of the user data stored in the succeeding mem- 
again functioning as **master" memory means and the ^ry becomes definitive (available for copying) for the 
further memory means (VLR) functioning as "slave" 45 ^^r after correctness has been confirmed and the cop- 
memory means. The base memory means (HLR) there- i^d portion of the user data stored in the other memory 
fore always function as "master" memory means, while becomes negligible(unavailable for copying), as a 
all the other memory means (VLRs) always fimction as result of which the required data are no longer copied 
"slave" memory means, at least as regards this user. over the generally large distance between the base 

A known method of this type has the disadvantage 50 memory and the succeeding memory but over the gen- 

that some user data are present in two different memo- erally smaller distance between the other memory and 

ries while the user is located in another work area asso- the succeeding memory which increases both the reli- 

ciated with an exchange other than the base exchange, ability and the efficiency of the method according to the 

which takes up memory capacity and consequently inveation compared with the known method, 

reduces the efficiency of the method, and that, if the 55 The present invention is based on the insight that the 

user continues to. move to successive work areas, the storing of the same data in two different memories is 

data required have to be copied over ever greater dis- inefficient and that the reliabihty and the efficiency of a 

tances from the base memory means (HLR) to succeed- method is increased if the data are copied over a smaller 

ing memory means (VLRs), which reduces both the distance, which is achieved by storing data in a memory 

reliability and the efficiency of the method. 60 situated as near to the user as possible. 

A first embodiment of the method according to the 

SUMMARY OF TOE INVENTION ^ characte^rrt.Tr?vent of 

The object of the invention is, inter alia, to provide a incorrectness of the copied portion of the user data 

more efficient and more reliable method. stored in the other memory with respect to the user data 

For this purpose, the method according to the inven- 65 stored in the base memory, the largest portion of the 

tion has the characteristic that, if the user moves from user data stored in the base memory is copied once more 

the base work area to the other work area, a coupling is and is stored via the coupling in the other memory, 

effected between the base memory and the other mem- This increases the reliability of the method further. 
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A second embodiment of the method according to 
the invention has the characteristic that user data not 
belonging to the largest portion and stored in the base 
memory comprises information relating to a location of 
the other exchange. 

As a result of storing the location of the other ex- 
change in the uncopied portion of the user data in the 
base memory, where the user is located is permanently 
known in the base exchange, which promotes the reli- 
ability. 

A third embodiment of the method according to the 
invention has the characteristic that the user data not 
belonging to the largest portion and stored in the base 
memory comprises information relating to a protection 
of the user. 

As a result of storing information relating to the pro- 
tection of the user in the imcopied portion of the user 
data in the base memory, this information is perma- 
nently available at one point and it is not copied and 



^- p-v*»v>WU »U1 

protection of the user and consequently the reliability. 

A fourth embodiment of the method according to the 
invention has the characteristic that user data not be- 
longing to the copied portion and stored in the other 



A sixth embodiment of the method according to the 
invention has the characteristic that, in the event of 
incorrectness of the copied portion of the user data 
stored in the second further memory with respect to the 

5 user data stored in the first further memory, the further 
largest portion of the user data stored in the first finther 
memory is copied once again and is stored via the fur- 
ther coupling in the second further memory. 
This increases the reliability of the method still fnr- 

JO ther. 

A seventh embodiment of the method according to 
the invention has the characteristic that the information 
belonging to the user data stored in the base memory 
and relating to a location of the other exchange is re- 
placed by information relating to a location of the sec- 
ond further exchange. 

As a result of storing the location of the second fur- 
ther exchange in the uncopied portion of the user data 
in the base memory, where the user is located is again 



15 



transmitted to another memory, which pronTotes the 20 "^"L^ ^"^"^^ ? 

protection of the user and con^lmenav the reliabilitv It^^^^.^^^^J^ ^^^^^SC which again 



memory comprises information relating to this other 25 ,>,?^i^°^ T ^ ^/f ^- 

exchanee. ® ample, the other work area) is to be understood citiier as 



promotes the reHabiUty. 

It is pointed out that the movement of a user from the 
first further work area (such as, for example, the base 
area) to the second further work area (such as, for ex- 



exchange. 

As a result of storing the information relating to the 
other exchange in the uncopied portion of the user data 
in the other memory, this information is available only 
at that point where the information is necessary and the 30 
base memory and succeeding memory are not encum- 
bered therewith, which promotes the efficiency. 

A fifth embodiment of the method according to the 
invention has the characteristic that, in the event of a 
further movement of the user from a first further work 35 
area associated with a first further exchange which 
comprises a first further memory to a second further 
work area associated with a second further exchange 
which comprises second further memory, a fiirther 
coupling is effected between the first further memory 40 
and the second further memory, a further largest por* 
tion of the user data stored in the first further memory 
being copied and stored via the further coupling in the 
second further memory, and the user data stored in the 
first further memory means being definitive (available 45 
for copying) for the user and changes to be made in the 
user data taking place both in the first further memory 
and in the second fUrther memory, the copied portion of 
the user data stored in the second further memory then 
being inspected for correctness with respect to the user 50 
data stored in the first further memory, whereafter, in 
the event of correctness, the portion of the user data 
stored in the second further memory becomes definitive 
(available for copying) for the user and changes to be 
made in the data take place only in the second further 55 
memory, the copied portion of the user data stored in 
the further memory becoming negligible (unavailable 
for copying). 

This embodiment illustrates that, if the user has al- 
ready moved from the base work area to the first fur- 60 
ther work area and the largest portion of the user data 
have been copied and stored in the first further mem- 
ory, and if the user then moves to the second further 
work area, the largest portion of the user data is in that 
case copied from the first further memory into the sec- 65 
ond further memory, which will generally take place 
over a smaller distance than if copying had to take place 
from the base memory. 



the movement of the user with his own terminal (such 
as in the case of mobile radial) or the movement of the 
user to another terminal (such as in the case of the fol- 
low-me feature). The movement of the user results in a 
movement of the largest portion of his user data to a 
new location but does not automatically imply that he 
also actually starts to communicate from his new loca- 
tion. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG, 1 shows a flow chart of the method according 
to the invention. 

DETAILED DESCRIPTION 

In tiie flow chart shown in FIG. 1, the blocks have 
the following meaning: 



block 

1 user i n dicat e s that he is moving from 
an old work area to a oew work area 

2 cooplicg is effected between old memory 
means and new memory means 

3 user data in the old memory means arc examined 
for the necessity to be copied: 

no, to block 4 
yes, to block 5 

4 the portion of the user data present in the old 
memory means and not to be copied b or has been 
separated from t2ie portion to be copied 

and is not copied; this refers, for example, 
to data relating to an oM exchange belonging to the 
■ old work area or, if the old work areas a 
base work area and an old exchange is a base 
exchange^ data relating to a location of the new 
exchange or relating to a protection of the user 
3 the portion to be copied of the user data 

present in the old memory means is copied and 
transmitted to the new memory means 

6 the copied portion of the user data is 
eiamined for changes to.be made: 
yes, to block 7 

00, to block 8 

7 the changes to be made are tn«H^ is the copied 

portion of the user data present In the old memory means; 
back to block S (this has the consequence that 
the portion to be copied of the user data present 
in the old memory means is copied once 
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block meaning 



and retransmission become superfluous (block 7). The 
user data stored in the new memory means are then 
examined for correctness (block 8). This can be done by 
comparing them with the data present in the old mem- 
ory means or by means of error-detecting and/or error- 
correcting codes. If correctness is not confirmed, the 
copied portion of the data stored in the old memory 
means is recopied and retransmitted (block 5) and if 
correctness is in fact confirmed, said iiser data stored in 
the new memory means become definitive for the user 
and are supplemented with information relating to said 
new exchange (block 9). Furthermore, the copied por- 
tion of the user data present in the old memory means 
becomes n^gible (unavailable for copying), which 
implies that they can be overwritten and that the mem- 
ory capacity used therefor can be reused and the infor- 
mation belonging to the user data present in the base 
memory means and relating to a location of the old 
exchange is replaced by information relating to a loca- 
tion of the new exchange. 

The user data stored in the new memory means are 
then examined for changes to be made (block 10). If 
there are changes to be made, these are only made in the 
user data stored in the new memory means (block 11 ) 
and, fiirthermore, whether the user wishes to move 
onoe again is examined again (block 12). If this is so, a 
further coupling is made fix)m the new memory means 
The method according to the flowchart shown in to yet newer memory means (block 2), etc., and if 
FIG. 1 proceeds as follows. As soon as a user indicates 30 this is not so, a communication can be started (block 13). 
that he is moving from an old work area (such as his 



again and is transmitted to the new memory means; 
of oovxse, it is also possible to maJce the chinges 
in both memory means, as a result 
of which reoewed copying and transmitdng ' 
becomes soperfluous) 

the user data stored in the new memory means arc 
examined for co r^fc- ? ^iw t 
no, back to block S 
yes, to block 9 

the oser data stored in the new memory means 
become definitive for the user and are sapplemcnted 
by information relating to the new 
exchange, the copied portion of the user data 
present b the old memory means becomes negligible 
and informadon belonging to user 
data present in the base memory means and relating 
to a location of the old exchange b replaced by 
infbrmaiion relating to a location of 
the new exchange 

the user data stored in the new memory 
means arc rTnmfnfrl for changes to be made: 
yes, to block 1 1 
no, to block 12 

the changes to be made are made in the user 
data present in the new memory means; 
back to bkx;k 10 

whether the user wishes to move again is examined: 
yesi, back to block 2 
no, to block 13 
communication can be started 
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base work area or a flrst further work area) to a new 
work area (such as another work area or a second fur- 
ther work area) (block 1 ), a coupling is effected be- 
tween old memory means (such as base memory means 35 
or first further memory ) and new memory means (such 
as other memory means or second further memory 
means) (block 2). This indication takes place in the case 
of mobile radio as a result of the user and his terminal 
being located between two base stations and the sue- 40 
ceeding base station measuring a stronger radio signal 
than the preceding base station and it takes place in the 
case of normal telephony as a result of the user switch- 
ing on a follow-me feature or informing a succeeding 
telephone set of his presence, for example, by means of 45 
a chip card. User data in the old memory means are then 
examined for the need to be copied (block 3). If the old 
memory means are the base memory means, die infor- 
mation relating to the location of the new exchange and 
relating to the protection of the user, for example, will 50 
not need to be copied and if the old memory means, are 
the first further memory means, the information relating 
to the first further exchange associated therewith, for 
example, will not need to be copied. In this way, the 
portion of the user data not to be copied in the old 55 
memory means is or has been separated from the por- 
tion which has indeed to be copied (block 4) and this 
lastmentioned portion is oopi^ and transmitted via the 
coupling to the new memory means (block 5). 

At this instant, all the user data are still located in the 60 
old memory means and the largest portion of said user 
data, which has been copied, is located in the new mem- 
ory means. This portion is examined for any changes to 
be made (block Q. If these exist, they are either made in 



I claim: 

1. A method of managing, in a telecommunication 
network, user data of a user who may move from a base 
work area associated with a base exchange which com- 
prises a base memory to another work area associated 
with another exchange which comprises another mem- 
ory, independent of whether or not the user is commu- 
nicating, comprising: 

effecting a coupling between the base memory and 
said another memory when the user moves from 
thfe base work area to said another work area; 

copying a largest portion of the user data stored in the 
base memory and storing the copied data in said 
another memory via the coupling, the user data 
stored in the base memory remaining available for 
further copying, and changes to be made in the user 
data taking place both in the base memory and in 
said another memory; 

inspecting the copied portion of the user data stored 
in said another memory for correctness with re- 
spect to the user data stored in the base memory; 
and 

in the event of correctness of the copied portion of 
the user data, making the portion of the user- data 
stored in said another memory available for further 
copying and wherein changes to be made in the 
data take place only in said another memory, the 
copied portion of the user data stored in the base 
memory thereafter becoming unavailable for fur- 
ther copying. 

2, The m^od of claim 1, wherein, in the event of 
incorrectness of the copied portion of the user data 
stored in said another memory with respect to the user 



the user data present in the old memory means, where- 65 data stored in the base monory, the largest portion of 



after these modified data are copied and transmitted 
again (block 5) or they are made both in the old and the 
new memory means, as a result of which the recopying 



the user data stored in the base memory is copied once, 
again and is stored via the coupling in said another 
memory. 
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3. The method of claim 2, wherein user data not be- 
longing to the largest portion and stored in the base 
memory comprises tnfonnation relating to a location of 
said another exchange. 

4. The method of claim 3, wherein the user data not 3 
belonging to the largest portion and stored in the base 
memory comprises information relating to a protection 
of the user. 

5. The method of claim 3, wherein user data not be- 
longing to the copied portion and stored in said another 10 
memory comprises information relating to said another 
exchange. 

6. The method of claim 3, wherda: 

in the event of a further movement of the user from a 
first further work area associated with a first for- 15 
ther exchange which comprises a first further 
memory to a second further work area associated 
with a second further exchange which comprise a 
second further memory, further comprising: 

effecting a further coupling means between the first 20 

' further memory and the second further memory; 

copying a further largest portion of the user data 
stored in the first further memory and storing said 
copied further largest portion of the user data via 
the further coupling m the second further memory, 25 
the user data stored in the first further memory 
being available for further copying and changes to 
be made in the user data taking place both in the 
first further memory and in the second further 
memory; 30 

inspecting the copied portion of the user data stored 
in the second farther memory for correctness with 
respect to the user data stored in the first further 
memory; and 

in the event of correctness of the inspected user data 35 
stored in the second further memory, the portion of 
the user data stored in the second further memory 
becoming available for further copying and 
wherein changes to be made in the data take place 
only in the second further memory, the copied 40 
portion of the user data stored in the first further 
memory becoming available for further copying. 

7. The method according to claim 6, wherein, in the 
event of incorrectness of the copied portion of the user 



8 

data stored m the second further memory with respect 
to the user data stored in the first further memory, the 
further largest portion of the user data stored in the first 
further memory is copied once again and stored via the 
further couplmg in the second further memory. 

8. The method according to claim 7, wherein the 
information belonging to the user data stored in the base 
memory and relating to a location of said another ex- 
change is replaced by the information relating to a loca- 
tion of the second further exchange. 

9. The method according to claim 6, wherein the 
information belonging to the user data stored in the base 
memory and relating to a location of said another ex- 
change is replaced by the information relating to a loca- 
tion of the second further exchange. 

10. The method of claim 2, wherein the user data not 
belonging to the largest portion and stored in the base 
memory comprises information relating to a protection 
of the user. 

11. The method of claim 2, wherein user data not 
belonging to the copied portion and stored in said an- 
other memory comprises information relating to said 
another exchange. 

12. The method of claim 1, wherein user data not 
belonging to the largest portion and stored m the base 
memory comprises information relating to a location of 
said another exchange. 

13. The method of claim 12, wherein the user data not 
belonging to the largest portion and stored in the base 
memory comprises information relating to a protection 
of the user. 

14. The method of claim 12, wherein user data not 
belonging to the copied portion and stored in said an- 
other memory comprises information relating to said 
another exchange. 

15. The method of claim 1, wherein the user data not 
belonging to the largest portion and stored in the base 
memory comprises information relating to a protection 
of the user. 

16. The method of claim 1, wherein user data not 
belongmg to the copied portion and stored in said an- 
other memory comprises information relating to said 
another exchange. 

* « * * ♦ 
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ABSTRACT 



An interleaved output queue is used as a high perfor- 
mance interface on a system bus for transferring infor- 
mation from a CPU to main memory. The queue is 
loaded on its input side with information that is bound 
for transmission from the CPU's cache to main memory. 
The queue itself is logically divided into those queue 
entry addresses which are either odd or even. On its 
output side, the queiie is unloaded by dual sets of unload 
circuitry, each of which accesses the information stored 
in either the odd or even queue entry addresses. Other 
select circuitry will alternate the transmission of infor- 
mation out of the two sets of unload circuitry to main 
memory. Each set of unload circuitry receives error 
information back from main memory during the time 
that the other unload circuitry is issuing a transaction. 
As a result, each set of unload circuitry is thereby in- 
formed whether its own previous transaction was error 
free, in which case it will transmit the information con- 
tained in its next queue entry address when it next issues 
a transaction^ or whether its previous transaction was 
not error free, in which case it will next have to resend 
the same information sent during the previous transac- 
tion. Consequently, transactions issued from the queue 
can proceed in an immediately successive sequence 
without waiting for the processing of error information 
from the immediately preceding transaction yet the 
queue can still recover form errors during high perfor- 
mance operation. 

27 Claims, 6 Drawing Sheets 
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prise several transactions. Tliereforc, a CPU which has 

SINGLE LOAD, MULTIPLE ISSUE QUEUE WITH a queue is capable of issuing multiple, immediately sue- 

ERROR RECOVERY CAPABILITY cessive transactions on the system bus as the queue is 

unloaded. 

FIELD OF THE INVENTION 5 Although an output queue can issue such multiple. 

Tlic present invention relates to an apparatus and a inunediaiely successive transactions, the conventional 

method for issuing high performance transfers of infor- approach to unloading a queue docs not take advantage 

mation from a central proccsang unit (CPU) within a of this capability because of the need of the system to 

computer to another location within that computer. recover from transmission and bus protocol errors, 

while still enabling the CPU to recover from transmis- Specifically, when a NACK results from a given trans- 

sion errors on a per transaction basis. action, the system must be able to reissue the transaction 

_ so that the information is once again sent from the trans- 

BACKGROUND OF THE INVENTION ^^^^ receiving node. Tliis conventional 

Computer systems are typically made up of multiple approach requires that no successive transaction be 

nodes which are connected together by a system bus commenced until there has been a successful comple- 

which carries information, such as commands, ad- tion of the preceding transaction, or until continuous 

dresses, data, and control signals between the nodes. unsuccessful attempts at transmission of the same infor* 

Ejiamples of the nodes which make up common com- mation result in a termixuting error, known as a "time 

puter configurations include central processing units. out**. 

main memory, and I/O adapters/controllers, which if, for example, the transmitting node were to corn- 
provide inteifaces to mass storage devices and net- mence a second transaction before receiving all of the 
works. When information is transferred over a bus the en-or information relating to the first transaction, and 
receiving node will inform the transmitting node the first transaction results in an error, the control sub- 
whether that information was corrupted during trans- sys^g^ ^^^^ transmitting node will have to jump back 
mission and whether there were any other protocol 25 and retransmit the first transaction after Uansmitting the 
errors resulting from the traiisaction. In ^rt, this is transaction. Then, assuming the second transac- 
accomplished by checking panty bits, which are trans- error-free, after the repeated execution of the first 
ferrcd along with the mfonnation. ^ , , ^ transaction, the comrol sub-system would have to jump 
When the receiving node has a^nfinned that the ,^ transaction, given that there is no 
transmission IS crror-frec, it will send back an acknowl- 30 ^^^^ transaction, which was sue- 
edgment CACK) signal to thetransmitiing node mdi- ^^^j ^ . ^^^^ . • ^^^^ ^ 
caung that the information is good. AUemai.vely ,f the ^^^^ J ^^^^^^^^^ 
information has been corrupted m transmission or any r . ted 

other bus protocol error occurs, the transmitting node ^"^P emen ea. 

Ar-v Ty.ie «t..« Arir is Accordmgly, imder the convenuonal unkjad method, 

does not return an ACK, Tms failure to return an ACK. J3 - 7 . • j • .l* r— . - 

. ^ K.. »h« *«rr«t^/Ni ic when the information contained in the first queue entry 

dunng the time period required by the bus protocol is . . . j . -n •* . i n -Tr 

logic^ly understood by the transmitting node as a "no- ^^^'^'fi ^^e system will wait until all of 

acknowledgment* (^NACK") signal indicating that the ^*»,e ACK*s have been returned before transmitting the 

information is not good, and the transmitting node must ">forrnation contamed in the second queue entry ad- 

once again transmit that information. 40 dress. If, however a NACK is retunied. the transaction 

There is some time delay between the completed « repeated. The disadvanuge of this approachjhow- 

transmission of the information to the receiving node. ever is that system performance is degraded b«:ai^ 

and the determination by the receiving node whether the further unloading of the qweuc is st^ cd while the 

the information was corrupted during transmission and « forced to wait for the ACK s or NACK s relat- 

whether there were any other protocol errors. This is 45 ing to a given transaction to be returned. This disadvan- 

because there must be some time for the information to tage is rather significant in light of the contmumg trend 
be checked against the parity bit by the bus interface of for the speed of CPU's to mcrease, which thereby 

the receiving node. Thus, if the transmitting node sends Peaces a greater demand on system mterconnects to 

information during a first bus cycle, it is common for «sue transactions in an immediately successive order, 

that node to wait some number of bus cycles before it 50 On the other hand, if the output queue's abUiiy to issue 

has an ACK or NACK informing it whether that infor- multiple, immediately successive transactions is to be 

mation is good. ^he design of the unload system for the queue must 

System designs known in the computer architecture be capable of recovering from errors when the original 

art often employ the use of queues for the temporary or subsequently attempted transactions result in at least 

storage of information which is bound for transmission 55 one NACK. Such error recovery capabUity is a proto- 

from one node to another. In a typical implementation, col requirement of some advanced system buses, 

a queue is positioned between a CPU and the interface In accordance with an aspect of the present inven- 

to the system bus. The CPU will load the queue with lion, there is provided a single output queue which is 

information that is to be transferred from the CPU to loaded through one set of load circuitry which operates 

main memory. An advantage of using the queue in this 60 at the same speed as the CPU. 

fashion is that it enhances CPU performance by allow- In another aspect of the invention the single output 

ing the CPU to continue processing additional opera- queue is logically divided into two separate logical 

tions before the actual completion of the transfer to queues; the two logical queues being made up of those 

main memory. In addition, in the typical implementa- physical queue entry address locations that are even, 

tion the loading of the queue takes place at the same rate 63 and those physical queue entry address locations that 

at which the CPU operates, which is necessary for are odd. 

maximum performance. The amount of information Another aspect of the invention is to access the odd 

stored by a output queue is usually sufTicient to com- and even queue entry addresses through two separate 



09/21/2003, EAST Version: 1.04.0000 



5,265 

3 

sets of unload circuitry, which thereby permits rapid 
unloading of the queue and enhances system perfor- 
mance. 

Still another aspect of the invention is to alternate 
between the unloading of odd and even queue entry 5 
addresses so that while one transaction is underway, the 
immediately preceding transaction can check iu own 
error information to determine whether it should rescnd 
the information in the same queue entry address again, 
or proceed to the next queue entry address. Moreover, 10 
this determination by the first half of the logical queue 
is done independent of the transaction issuance state of 
the other half of the logical queue. Therefore, the dual 
access system permits unloading at high performance, 
but also provides a sophisticated method of error recov- 15 
ery which is practical to implement. 

SUMMARY OF THE INVENTION 

In accordance with the present invention, an output 
queue for writing information from a CPU to main ZD 
memory is logically divided into its odd and even queue 
entry addresses. The queue has a single load path on its 
input side, allowing the loading speed to be the same as 
the operating speed of the CPU. Associated with the 
queue is a dual set of unload circuitry, one set of which 25 
accesses the odd queue entry addresses and the other 
accesses the even queue entry addresses. Further associ- 
ated with each set of unload circuitry is an address 
pointer and error response circuitry. Ultimately, the 
unload circuitry is connected to the system bus for 30 
transmitting information out of the queue to main mem- 
ory. 

After the queue is loaded with information it is emp- 
tied through the unload circuiuy. Specifically, the even 
address circuitry accesses the information stored in the 35 
first even queue entry address, and transmits thai infor- 
mation to main memory. Immediately thereafter, the 
odd address circuitry accesses the information stored in 
the first odd queue entry address and transmits that 
information to main memory. While the second transac- 40 
tion is underway, parity and protocol error information 
relating to the first transaction is evaluated by the error 
response circuitry to determine whether the address 
pointer for the even queue entry addresses should be 
advanced to the next even address of the queue. If an 45 
error in the first transaction is detected, the address 
pointer will be stalled so that immediately following the 
completion of the second transaction, the first transac- 
tion will be repeated. 

Likewise, while the information in the fu-st even 50 
queue entry address is being retransmitted, parity and 
protocol error information relating to the second trans- 
action is evaluated by the error response circuitry to 
determine whether the address pointer for the odd 
queue entry addresses should be advanced to the next 55 
odd queue entry address. Assuming thai the second 
transaction was error-free, the odd address counter will 
advance so that the second odd queue entry address 
information will be transmitted as soon as the retrans- 
mission of the first even queue entry address is com- 60 
pleted. 

The unloading of the queue will generally continue 
this ordered sequence of accessing odd and even queue 
entry addresses in like fashion until the queue is empty. 
Thus, the queue functions as a high performance inter- 65 
face between the CPU and main memory, being loaded 
on its input side at the same operating speed as the CPU. 
Moreover, due lo its ability to reissue a transaction 
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which resulted in an error, the queue can contintie high 
performance operation even under error recovery con- 
ditions. 

BRIEF DESCRIPTION OF THE DRAWING 

FIG. 1 is a general system overview showing multi- 
ple CPU modules each having a write-back queue, mul- 
tiple I/O modules, and multiple main memory modules 
all linked together by a system bus. 

FIG. 2 is an illustration of the output queue with its 
related load and unload circuitry. 

FIG. 3 is a more detailed illustration of the output 
queue, using a three dimensiona] characterization to 
show the multiple blocks of information stored in the 
individual queue entry address locations. 

FIG. 4 is a representation of the D latches that com- 
prise the output queue storage locations. 

FIG. 5A is a timing diagram that shows the perfor- 
naance of an output queue using a conventional, single 
unload approach, while FIG. 5B is a similar timing 
diagram that shows the performance of an output queue 
using the dual unload approach of the present invention. 

FIG. 6 illustrates an alternate embodiment of the 
present invention, showing multiple, dual unload output 
queues linked together to achieve even higher transac- 
tion performance than one, dual unload output queue. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

The present invention relates to a queue which is 
loaded through one set of load circuitry which operates 
at the high performance speed of the CPU. and un- 
loaded through multiple sets of unload circuitry. The 
exact number of sets of unload circuitry is determined 
by logically dividing the queue into some number of 
interleaved, queue entry addresses and issuing transac- 
tions from those addresses in an ordered sequence. Con- 
sequently, through using this interleaved, transaction 
issuance approach the queue provides a high perfor- 
mance interface with the system bus. 

In the preferred embodiment of the invention the 
queue is designed to write information from a CPU to 
main memory through the use of a single, circular queue 
with dual unload circuitry. A circular queue is a queue 
which is loaded with information, and as it is unloaded, 
the queue entry addresses which have been emptied are 
then available for additional information to be loaded 
again. The use of this queue substantially improves 
system performance by having transactions on the sys- 
tem bus proceed one right after the other, without any 
given transaction having to be acknowledged as error- 
free before commencing the next transaction. The issu- 
ance of transactions in this manner, namely commenc- 
ing one transaction before receiving back complete 
error information regarding the immediately preceding 
transaction, shall hereafter be defined as "multiple, im- 
mediately successive transactions". In addition, even 
though transactions can proceed in this tightly ordered 
sequence, the interleaved write-back approach is capa- 
ble of recovering from transmission errors using a so- 
phisticated, but practical to implement, error recovery 
scheme. 

Referring now to the drawing, FIG. 1 shows a com- 
puter system, including multiple, identical CPU riKxl- 
ules 20(a-n), multiple, identical input/output (I/O) mod- 
ules 22(a-n), and multiple, identical main memory mod- 
ules 24(a-n). The illustration shows *n' number of CPU 
modules, I/O modules, and main memory modules be- 
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cause the exact number of CPU, I/O, and main memory 
modules can vary according to system use and perfor- 
mance objectives. These modules are linked together 
through a system bus 26, which in the preferred em- 
bodiment is Digital Equipment Corporation*s XMI bus. 5 
Also shown in FIG. 1, the CPU module 20 has a CPU 
28 which is connected to a cache 29, which, in turn, is 
connected to the output queue 30. The queue 30 is posi- 
tioned between the cache 29 and the system bus 26. 
AJthough not shown in FIG. 1, in a multiprocessor 10 
environment all of the CPU modules would be identi- 
cally conftgured with their own CPU's, caches and 
queues. 

In FIG. 2, the output queue 30 is shown along with 
the block circuitry that loads it on its input side and 15 
unloads it on its output side. Specifically, information, 
such as commands, addresses, and data, are sent from 
the cache 29 and are loaded into the queue 30 through 
the demultiplexor 32, which has an input and an output 
side and which is controlled by the load control 34. The 20 
demultiplexor 32 and Its associated load control 34 are 
responsible for filling the various queue entry addresses 
of the queue 30 with commands, addresses, and data 
which are sent by the cache 29 to the main memory 
module 24. The queue 30 has eight queue entry address 25 
locations (Q0-Q7) and each queue entry address loca- 
tion is capable of holding five quad words (1 quad- 
word=64 bits), one of those quad words typically being 
a command (C) together with an address (A) and the 
other four quadwords being data (DO, Dl, D2, D3). 30 
Although in this implementation of the invention the 
queue 30 has eight queue entry addresses and five quad- 
words in each queue entry address, the actual number of 
queue entry addresses and words, and the actual word 
size may vary depending upon design factors and goals. 35 

Although the conventional method of unloading an 
output queue would involve the use of a single multi- 
plexer to empty the various queue entry address loca- 
tions, FIG. 2 shows the queue 30 connected to two 
multiplexers, even queue multiplexer 36 and odd queue 40 
multiplexer 38. Through the queue multiplexers 36, 38 
the queue 30 can be accessed at multiple points for the 
purpose of unloading the information contained its eight 
queue entry address locations. Even though the queue 
30 is physically only one queue, it logically has the same 45 
furK:iionality as two separate queues the two logical 
queues being deftned by the queue entry addresses 
which are either odd or even. The unloading of the 
queue 30 will proceed with the emptying of first an 
even queue entry address, such as QO, then an odd 50 
queue entry address, such as Ql, then an even queue 
entry address, such as Q2, and so on. Furthermore, 
other implementations of the invention can divide the 
queue 30 into multiples of any number depending upon 
design and performance factors; however, additional 55 
queue divisions would necessitate the implementation of 
additional unloading multiplexers, similar to the queue 
multiplexers 36, 38, and their related unload control 
circuitry. 

The input sides of the queue multiplexers 36, 38 are 60 
connected to the output side of the queue 30, and can 
unload the queue 30 by accessing its eight queue entry 
address locations. As shown in FIG. 2, even queue 
multiplexer 36 is connected to the even queue entry 
addresses of queue 30 (QO, Q2, Q4, Q6), and odd queue 65 
multiplexer 38 is connected to the odd queue entry 
addresses of queue 30 (Ql, Q3, Q5. Q7). Even queue 
multiplexer 36 is controlled by even unload control 40 
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which is a block representation which includes the con- 
trol circuitry that determines which queue entry ad- 
dress of the queue 30 even queue multiplexer 36 will be 
accessing during any given transaction. This control 
circuitry is comprised of an address pointer which ena- 
bles even queue multiplexer 36 to access each of the 
even queue entry addresses within queue 30. The incre- 
menting of the address pointer is determined by error 
response circuitry, which is also contained in even un- 
load control 40. 

The error response circuitry receives and rcacu to 
parity and protocol error information, the ACK*s and 
NACK's, resulting from the transmission of information 
to the main memory 24. After receiving and processing 
this information the error response circuitry detennines 
whether the address pointer for even queue multiplexer 
36 should be advanced to the next address, or should be 
stalled at the same address because an error has necessi- 
uted a retransmission of the information in the same 
queue entry address location. Therefore, even unload 
control 40 is connected to the system bus 26 through the 
error state lines 42. The error state lines 42 carry parity 
and protocol error information back to even unload 
control 40 in the form of ACK and NACK signals. The 
effects that this error information has on accessing in- 
formation in the queue 30 by queue multiplexers 36, 38 
shall be further detailed in connection with the discus- 
sion of FIGS. 5A and SB, below. 

As shown in FIG. 2, odd queue multiplexer 38 is 
similarly controlled by odd unload control 44, the con- 
tents and function of which are virtually identical to 
even unload control 40, except that it determines which 
odd queue entry addresses of queue 30 odd queue multi- 
plexer 38 will be accessing during any given transac- 
tion. Odd unload control 44 contains an address pointer 
which is controlled by error response circuitry which is 
similar to the address pointer and error response cir- 
cuitry of even unload control 40. In addition, odd un- 
load conuol 44 is connected to the system bus 26 
through the error state lines 42 so that the error re- 
sponse circuitry can determine whether the address 
pointer should be incremented after a given transaction, 
or whether the same transaction will have to be re- 
peated. 

Finally, in FIG. 2 the output sides of the queue multi- 
plexers 36, 38 are connected to the input sides of the 
select multiplexer 46, while the output side of select 
multiplexer 46 is connected to the system bus 26. The 
select multiplexer 46 is controlled by the select control 
48, and together they determine which of the two queue 
multiplexers 36, 38 transmits its information on the sys- 
tem bus 26. Under most circumstances, the select con- 
trol 48 causes the select multiplexer 46 to toggle be- 
tweetr the queue multiplexers 36. 38 so that the select 
multiplexer 46 alternates between transmitting the in- 
formation contained in an even queue entry address, 
then the information contained in an odd queue entry 
address, then an even queue entry address, and so on. In 
the meanwhile, the queue multiplexers 36, 38 and their 
related unload controls 40, 44 are responsible for deter- 
mining which even or odd queue entry address, respec- 
tively, is transmitted to the select multiplexer 46. 

FIG. 3 shows the same essential features of the queue 
30 and its related loading and unloading circuitry as 
shown in FIG. 2; however, it docs so with enhanced 
detail. The three dimensional character of the drawing 
is intended to illustrate that each of the eight queue 
entry addresses of the queue 30 is actually made up of 
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multiple blocks of information. That infonnation. in the gram are bus cycle numbers, below which ts shown the 

form of commands, addresses, and data, comprises information that is transmitted during those bus cycles 

blocks of 64 bits each, and each transaction has associ- Assuming that the conventional queue is prepared to 

atcd error status information which is generally desig- transmit information to main memory, during cycle #0, 

nated by the letter "E". The dcmultiplexor 32 is shown 5 the CPU module will arbitrate for the bus. In this cxam- 

in three dimensional form because it must not only be pie, access to the system bus has been granted during 

able to distinguish between the eight queue entry ad- cycle pfh the granting of access having been deter, 

dress locations of the queue 30 when it is filling it with mined by the priority level assigned to the CPU module 

information, but it also must be able to distinguish be- through the arbitration scheme. During cyde #1 the 

tween the five blocks of information at any given queue 10 queue will irananil the C/A from the first block of QO, 

entry address. Similarly, the queue multiplexers 36, 38 and during the following four cycles. ##2-5. it will 

are also shown three dimensionally because they, too, transmit D0-D3 from the remaining four bjocks of QO. 

must each be able to distinguish between not only their According to the XMI bus protocol, ACK and 

four respective queue entry addresses when unloading, NACK signals from main memory are not received by 

but also the five blocks of information at any given 15 the CPU module untU two cycles after a given blodt of 

queue entry address. information is sent by the CPU module. Therefore. 

All of the actual connections between the demulti- during cycle #3 the queue is sending Dl to main mem- 
plexor 32 and the queue 30. and the queue 30 and the ory over the data lines of the bus, while main memory 
queue multiplexer? 36, 38 arc not shown in FIG. 3 in the is returning an ACK. which relates to the C/A, on the 
interest of preserving the clarity of the illustration. In an 20 separate error stale lines. During cycle #4 the queue is 
actual implementation of the invention, those connec- sending D2 to main memory, while main memory is 
tions, however, would be made in a fashion similar to returning an ACK which relates to DO. and so on with 
the connections that are shown on FIG. 3. all of those the succeeding cycles. Accordingly, information and 
connections being shown generally on FIG. 2. error stete signals may be traveling in opposite direc- 

Rcfcrring to FIG. 4. the individual memory cells of 25 tions on the bus at any given time because they are on 

the queue 30 are made up of D latches 50 which are distinct lines. 

enabled using a signal on the clock 52; however, in Still referring to FIG. 5A, the performance limita- 

another embodiment of the invention, the individual tions of emptying the conventional output queue can be 

memory cells of the queue 30 can be made up of any seen. Specifically, after the transmission of the C/A and 

storage means which meets the performance needs of 30 D0-D3 during cycles ##1-5, which together com- 

the system. prises a single transaction, the complete error state in- 

FIG. 4 shows the five infonnation blocks, each con- formation on that transaction will not be received until 

laining 64 bits, which make up the information stored at cycle #7. During cycles ##8-10 the error logic for the 

one single queue entry address of the queue 30. in this conventional queue will have to process the error infor- 

insunce QO is shown. In the preferred embodiment, the 35 mation from that first transaction to determine which 

first block contains a command (C), such as a write queue entry address will next be accessed. Given that a 

command, and an address (A), and the next four blocks NACK was returned during cycle #7, meaning that 

contain data (DO, Dl. D2. D3). The function of the there was an error resulting from D3. the queue will 

command is to provide main memory with an instruc- have to retransmit all of the information contained in 

tion as to what it is supposed to do with the data which 40 QO. Therefore, during cycle #11 the CPU module arbi- 

will be following the C/A block, while the address irates for and. according to the example, is granted 

informs main memory what address is to be associated access to the bus. Thereafter, the information contained 

with the command and the dau. Thus, a write com- in QO is retransmitted commencing with cycle #12. 

mand would instruct main memory to write the four Because of the delay in the receipt of error inforraa- 

following blocks of dau beginning with the address sent 45 tion and because of the need to be able to recover from 

along with the write command. errors, the conventional approach is to wait until all of 

When a queue entry address is accessed, the given the acknowledgment signals have been received before 

queue multiplexer, 36 or 38, will first send the C/A again arbitrating for the bus and attempting the next 

block stored at that queue entry address as a complete transaction. As a result, in the conventional method the 

block of 64 bits. The queue 30 will next send DO as 50 issuance of the next transaction is simply stalled until 

another complete block of 64 bits, followed by Dl. and the preceding transaction is successfully completed, or 

so on until all five blocks at that given queue entry a time out error terminates further attempts at transmit- 

address have been sent. The transmission of these five ting the transaction. On the other hand, it is clear thai 

related blocks of information comprise a single "trans- the overall performance of the unloading system is 

action". 55 impaired in that four cycles per transaction are wasted 

According to the XMI bus protocol, after the trans- while the system waits for the error logic to process the 

mission of the first block of information, which is made ACK's/NACK's. 

up of the C/A, the main memory 24 must return an The unloading performance of the queue 30 can be 

ACK signal to confirm that that block of information substantially improved through the use of the present 

was without error. If no such acknowledgment is re- 60 invention. Referring now lo FIG. 5B, a timing diagram 
ceived during the assigned time period, the absence of similar to FIG. 5 A is shown; however, in this instance, 

the confirmation is interpreted as a NACK. which the timing diagram rcfiects the dual unload approach 

means there has been an transient error in transmission using the queue 30, the queue multiplexers 36, 38, the 
or some protocol error and the information must be unload controls 40, 44, the select multiplexer 46 and the 

retransmitted. 65 select control 48, as discussed above in connection with 

Referring now to FIG. 5A, a timing diagram is FIGS. 2 and 3. 

shown for an output queue which uses the conven- As shown in FIG. 5B, during cycle #0 the CPU 
tional. single unload method. Along the top of the dia- module 20 arbitrates for and is granted access to the 



09/21/2003, EAST 



Version: 1 . 04 . 0000 



5,265, 

9 

system bus 26. Simiiar to the example illustrated 
through FIG. 5A, the queue 30 likewise commences the 
unloading of the first even queue entry address, QO, 
during cycle #1 by transmitting the first block of infor- 
mation, namely C/A, followed by the four data blocks, S 
D0-D3, during the four succeeding clock cycles. 
Again, acknowledgments relating to any given block of 
information will be received by the even unload control 
40 two cycles after the information is sent. Consistent 
with the example given in FIG. 5B, the transmission of 10 
D3 has resulted in a NACK, which is received during 
cycle #7. In this instance, however, during cycle #6 the 
CPU module 20 has arbitrated for and been granted 
access to the system bus 26 thus enabling it to com* 
mence a second transaction, namely the unloading of IS 
the fust odd queue entry address, Ql, commencing with 
cycle #7. 

It is important to note that the XMI bus protocol 
requires at least one bus cycle between transactions 
during which no command, address, or data infonna* 20 
tion is on the bus, so that other nodes may attempt to 
gain bus access according to their respective orders of 
priority. In the example illustrated in this timing dia- 
gram, cycle }t^6 is this inactive cycle, and also in this 
example no other node has been granted access which 2S 
delays the continued write-back operation of the queue 
30. 

Still referring to FIG. SB, before even unload control 
40 receives back all of the error information relating to 
the first transaction, odd queue multiplexer 38, which 30 
accesses the odd queue entry addresses, will transmit 
the information contained in Ql commencing with 
cycle #7. While this second transaction is underway, 
the error response circuitry contained in even unload 
control 40 will process the ACK's/NACK's relating to 35 
the first transaction, and thereby determine whether the 
address pointer for even queue multiplexer 36 should be 
advanced to point to the next even queue entry address 
of queue 30. If a NACK relating to the first transaction 
is received, as is shown in FIG. 5B during cycle #7, that 40 
address pointer will be stalled at QO. so that all of the 
information contained in QO is retransmitted, as is 
shown commencing with cycle #12. 

A NACK relating to any one of the five blocks com- 
prising the first transaction would have caused a re- 45 
transmission of QO; however, the latest returning error 
signal, namely the one returned during cycle #7, was 
chosen to illustrate that the system can recover from 
errors, with high performance, even if they are not 
detected until the last return cycle relating to a given 50 
transaction. 

Similarly, while the information contained in QO is 
retransmitted during cycles ##13-17, the error re- 
sponse circuitry contained in odd unload control 44 will 
process the ACK*s/NACK's relating to the second 55 
transaction, and thereby determine whether the address 
pointer for odd queue multiplexer 38 should be ad- 
vanced to point to the next odd queue entry address of 
queue 30. In accordance with the example shown in 
FIG. SB, the second transaction involving the transmis- 60 
sion of the information contained in Ql was error-free; 
alt of the ACK*s were received. Therefore, the odd 
address pointer wilt be incremented so that the fourth 
transaction will involve the transmission of the next odd 
queue entry address, Q3, as shown commencing with 65 
cycle #19. 

Assuming that the second attempt at transmitting the 
information contained in QO is error-free, the even ad- 



dress pointer will be similarly incremented during the 
fourth transaction, which involves the transmission of 
the information contained in Q3. so that it next points to 
Q2. After the completion of the transmission of the 
information in Q3, even queue multiplexer 36 will trans- 
mit the information contained in Q2. The alternating 
transmission of the information contained in the odd 
and even queue entry addresses of the queue 30 will 
continue in this ordered sequence until the queue 30 is 
emptied, or the last pending transaction times out as a 
result of repeated unsuccessful transmission attempts. 

As can be seen by comparing FIGS. 5A and SB, the 
present invention can almost double the transmission 
performance of the queue 30. Referring specifically to 
the transaction issuance of the single unload queue and 
the dual unload queue during cycles ##0-13, it can be 
seen thai by the time the single unload queue com- 
mences its second transaction, during cycle #12, the 
dual unload queue is very close to completing its second 
transaction, that second transaction actually being com- 
pleted during cycle #13. This very substantial perfor- 
mance improvement is a direct result of the ability of 
the queue 30 to be able to issue multiple, immediately 
successive transactions, while maintaining a high per- 
formance single load path to the queue 30. 

A further advantage of the present invention is that 
its application may be extended to include multiple 
output queues. As shown in FIG. 6, 'n' number of iden- 
tical output queues 30(a-n), with their related demulti- 
plexors 32(a-n), even queue multiplexers 36(a-n) and 
odd queue multiplexers 38(a-D), are connected to the 
select multiplexer 46. Although this drawing does not 
include the detail of FIOS. 2 and 3, each queue would 
be identical to the queue 30 and its related loading and 
unloading circuitry as shown and discussed in connec- 
tion with FIGS. 2 and 3. As can be seen, the present 
invention can be replicated any number of times if the 
performance goals of the system require very high 
speed transaction issuance. Such a design approach 
could yield significant benefits in a pipelined computer 
system where high transaction issuance bandwidth is 
needed. 

In addition, the present invention is not limited to the 
transmission of information between a CPU module and 
main memory, only. Rather the invention is applicable 
to transmissions between any high speed cache struc- 
tures on a pended bus. For example, an I/O module 
which is configured with a high speed cache could 
implement this unload approach to improve its perfor- 
mance. 

Accordingly, additional advantages and modifica- 
tions will readily occur to those skilled in the art The 
invention in its broader aspects is therefore not limited 
to thtf specific details, representative apparatus, and 
illustrative examples shown and described above. Thus, 
departures may be made from such details without de- 
parting from the spirit or scope of the invention. 

What is claimed is: 

1. An apparatus for issuing transactions from a first 
node within a computer system to a second node within 
a computer system, said apparatus comprising: 

a queue having a plurality of queue entry addresses 
for electronically storing information; 

a loading means for loading information into said 
queue, said loading means being adapted for elec- 
trical coupling with a first node such that with said 
loading means electrically coupled to a first node 
said loading means receives information sent from 
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a first node, and said loading means being electri- 
cally coupled with said queue such that said load- 
ing means loads information into said queue entry 
addresses; 

a first unloading means electrically coupled to a first 5 
group of queue entry addresses for transferring 
infonnation contained in said first group of queue 
entry addresses to a second node; 

a second unloading means electrically coupled to a 
second group of queue entry addresses, which are 10 
different from said first group of queue entry ad- 
dresses, for transferring information contained in 
said second group of queue entry addresses to a 
second node; and 

means for alternately enabling said first and second IS 
unloading means to issue multiple, immediately 
successive transactions from said queue, said en- 
abling means being adapted for coupling with a 
second node for receiving error state information 
from a second node, which indicates that a given 20 
transmission of data from a given queue entry ad- 
dress to a second node was erroneous, said error 
state information being used by said «iabling 
means to cause a data transmission which results in 
an error to be repealed. 25 

2. An apparatus as in claim 1, wherein said loading 
means includes a demultiplexor. 

3. An apparatus as in claim 1, wherein said first un- 
loading means includes a queue multiplexer. 

4. An apparatus as in claim 3, wherein said enabling 30 
means includes an address pointer for indicating which 
single queue entry address said multiplexor unloads 
during a given transaction. 

5. An apparatus as in claim 4, wherein said enabling 
means further includes error response circuitry, which 35 
is responsive to error state information from a second 
node for providing a signal to increment said address 
pointer if the immediately preceding transaction exe- 
cuted by said queue multiplexer did not result in an 
error. 40 

6. An apparatus as in claim 1, wherein said enabhng 
means include a select multiplexer which is electrically 
coupled to said first and second unloading means for 
determining which one of said first and second unload- 
ing means transmits information to a second node dur- 43 
ing any given transaction. 

T An apparatus for issuing transactions from a central 
processing unit (CPU) module within a computer sys- 
tem to a main memory, said apparatus comprising: 

a queue having a plurality of queue entry addresses 50 
for electronically storing infonnation; 

a loading means for loading information into said 
queue, said loading means being adapted for elec- 
trical coupling with a CPU module such that with 
said loading means electrically coupled to a CPU 55 
module said loading means receives information 
sent from a CPU module, and said loading means 
being electrically coupled with said queue such 
that said loading means loads information into said 
queue entry addresses; (C 

a first unloading means electrically coupled to a first 
group of queue entry addresses for transferring 
information contained in said first group of queue 
entry addresses to a main memory; 

a second unloading means electrically coupled to a 65 
second group of queue entry addresses, which are 
different from said fu^t group of queue entry ad- 
dresses for transferring information contained in 
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said second group of queue entry addresses to a 
main memory; and 
means for alternately enabling said first and second 
unloading means to issue multiple, inmiediately 
successive transactions from said queue, said en- 
abling means being adapted for coupling with a 
main memory for receiving error state infonnation 
from a main memory, which indicates that a given 
transniission of data from a given queue entry ad- 
dress to a main memory was erroneous, said error 
state information being used by said enabling 
means to cause a data transmission which results in 
an error to be repeated. 

8. An apparatus as in claim 7, wherein said loading 
means includes a dcmultiplexor. 

9. An apparatus as in claim 7, wherein said first un- 
loading means includes a queue multiplexer. 

10. An apparatus as in claim 9, wherein said enabling 
means includes an address pointer for indicating which 
single queue address said queue multiplexor unloads 
during a given transaction. 

11. An apparatus as in claim 10, wherein said enabling 
means further includes error response circuitry, which 
is responsive to error state information from a main 
memory for providing a signal to increment said ad- 
dress pointer if the immediately preceding transaction 
executed by said queue multiplexer did not result In an 
error. 

12. An apparatus as in claim 7, wherein said enabling 
means includes a select multiplexer which is electrically 
coupled to said first and second unloading means for 
determining which one of said first and second unload- 
ing means transmits information to a main memory 
during any given transaction. 

13. An apparatus for issuing write-back transactions 
from a central processing unit (CPU) module within a 
computer system to a main memory, said apparatus 
comprising: 

a write-back queue having a plurality of queue eritry 
addresses for electronically storing information, 
said queue entry addresses being logically divided 
into even and odd queue entry address locations; 

a loading means for loading information into said 
queue, said loading means being adapted for elec* 
trical coupling with a CPU module such that with 
said loading means electrically coupled to a CPU 
module said loading means receives information 
sent from a CPU module, and said loading means 
being electrically coupled with said queue such 
that said loading means loads information into said 
queue entry addresses; 

a first unloading means electrically coupled to said 
even queue entry addresses for transferring infor- 
mation contained in said even queue entry ad- 
dresses to a main memory; 

a second unloading means electrically coupled to said 
odd queue entry addresses for transferring informa- 
tion contained in said odd queue entry addresses to 
a main memory: and 

means for alternately enabling said first and second 
unloading means to issue multiple, immediately 
successive transactions from said queue, said en- 
abling means being adapted for coupling with a 
noain memory for receiving error state information 
from a main memory, which indicates that a given 
transmission of data from a given queue entry ad- 
dress was erroneous, said error state information 
being used by said enabling means to cause a data 
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transmission which results in an error to be re- 
peated. 

14. An apparatus as in claim 13, wherein said loading 
includes a dexnultiplexor. 

15. An apparatus as in claim 13. wherein said first ^ 
unloading means includes a queue multiplexer. 

16. An apparatus as in claim 15, wherein said enabling 
means includes an address pointer for indicating which 
single queue entry address said queue multiplexor un- 
loads during a given transaction. 

17. An apparatus as in claim 16, wherein said enabling 
means further includes error response circuitry, which 
b responsive to error information from a main memory 
for providing a signal to increment said address pointer is 
if the immediately preceding write-back transaction 
executed by said queue multiplexer did not resuh in an 
error. 

18. An apparatus as in claim 13, wherein said enabling 
means includes a select multiplexer which is electrically 20 
coupled to said unloading means for determining which 
one of said first and second unloading means transmits 
information to a main memory during any given trans- 
action. 

19. A method of unloading information from a fu^t 
node in a computer to a second node in a computer 
comprising the steps of: 

loading a queue, which has a plurality of queue entry 
addresses, with information from a first node; 

logically grouping all of said queue entry addresses 
into first and second queue entry address groups, 
the second group of queue entry address groups 
being different from the first; 

using first and second unloading means, which are 33 
each respectively coupled to said first and second 
queue entry address groups, to unload said queue 
by having said unloading means alternately trans- 
mit information out of their respective queue entry 
address groups in multiple, immediately successive 40 
transactions; and 

using error state information from a second node, 
which Indicates that a given transmission of data 
from a given queue entry address to a second node 
was erroneous, to cause a data transmission which 
results in an error to be repeated. 

20. The method as in claim 19, further comprising the 
step of: 

using an address pointer for indicating the single ^ 
queue entry address of a given address group 
which is unloaded during a given data transmis- 
sion, 

21. The method as in claim 20, funher comprising the 
step of: 55 

incrementing the address pointer to the next address 
if the error stale information from the previous 
data transmission out of the address group with 



which the pointer is associated did not indicate that 
the previous transmission was in error. 

22. A computer system comprising: 
first and second nodes. 

a queue having a plurality of queue entry addresses 
for electronically storing information which is 
transferred by said first node to said second node; 

a loading means for loading information into said 
queue, said loading means being electrically cou- 
pled with said first node for receiving information 
sent from said first node, and said loading means 
being electrically coupled with said queue for load- 
ing information into said queue entry addresses; 

a first unloading means electrically coupled to a fu^t 
group of queue entry address^ for transferring 
information contained in said first group of queue 
entry addresses to said second node; 

a second unloading means electrically coupled to a 
second group of queue entry addresses, which are 
different from said first group of queue entry ad- 
dresses, said second unloading means being for 
transferring information contained in said second 
group of queue entry addresses to said second 
node; 

said second node transmitting error state information 
to said first node indicating whether the transfer of 
any of information from said first node to said sec- 
ond node was in error; and 

means for alternately enabling said first and second 
unloading means to issue multiple, immediately 
successive transactions from the queue entry ad- 
dresses to which they are respectively coupled, 
said enabling means being coupled to said second 
node to receive the error state information, which 
is used by said enabling means to cause a transmis- 
sion which results in an error to be repeated. 

23. An apparatus as in claim 22. wherein said loading 
means includes a demultiplexor. 

24. An apparatus as in claim 22, wherein said first 
unloading means includes a queue multiplexer. 

25. An apparatus as in claim 24, wherein said enabling 
means includes an address pointer for indicating which 
single queue entry address said multiplexor unloads 
during a given transaction. 

26. An apparatus as in claim 25, wherein said enabling 
means further includes error response circuitry which is 
responsive to error state information from said second 
node for providing a signal to increment said address 
pointer if the immediately preceding transaction exe- 
cuted by said queue multiplexer did not result in an 
error. 

27. An apparatus as in claim 26, wherein said enabling 
means includes a select multiplexer which is electrically 
coupled to said unloading means for determining which 
one of said fu-si and second unloading means transmits 
information to said second node during any given trans- 
action. 
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machines and a central agent server present in another 
machine. The agent server comprises autonomous agents (5) 
that communicate via notifications, a storage layer and an 
agent machine comprising an execution engine, a commu- 
nication channel and two message queues for the 
notifications, a local queue (mqln) and an external queue 
(mqOut). The execution engine takes a notification in the 
local queue, determines and loads the corresponding agent 
for each notification, has the agent react to the notification, 
which agent can then change its status and/or send notifi- 
cations to the communication channel (2), which stores them 
in the local queue if they are addressed to a local agent and 
in the external queue if they are addressed to an agent of 
another server. 
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Network::HandleIn(); 
queueIn.Get(Not, agid); 
og=Agent::LoQd(agtd); 
og.React(Not); 
ag.Save(); 
queueln.PopQ; 
Commit(); 

Network::HandleOut(); 



FIG. I 



09/21/2003, EAST Version: 1.04.0000 



U.S. Patent Nov. 5, 2002 sheet 2 of 3 US 6,477,564 Bl 




FIG. 2 



09/21/2003, EAST Version: 1.04.0000 



U.S. Patent Nov. S, 2002 sheet 3 of 3 US 6,477,564 Bl 





class Agent { 




React (Notification) 


throw Exc; 


SaveO throw Exc; 




Load (Agentid) throw 


Exc; 


}; 





FIG. 3 



r 

class Channel { 

SendTo (Agentid, Notification) throw Exc; 
}; 



FIG. 4 




Class messageQueue { 
Push (Agentid, Notification) throw Exc; 
Get (Agentid, Notification) throw Exc; 
PopO throw Exc; 

}; 



FIG. 5 
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PROCESS FOR TRANSFORMENG AND 
ROUTING DATA BETWEEN AGENT 
SERVERS PRESENT IN SOME MACHINES 
AND A CENTRAL AGENT SERVER 
PRESENT IN ANOTHER MACHINE 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The invention relates to a process for transforming and 
routing data between agent servers present in some 
machines and a central agent server present in another 
machine. 

More specifically, the invention relates to a process for 
transforming and routing data between agent servers present 
in machines of production sites and a centra! agent server 
present in the machine of the decision-making base, for 
example within the framework of a centralized concentra- 
tion of data, also known as a Data Warehouse. 

2. Description of Related Art 

Data Warehousing is a computing concept aimed at con- 
centrating in a centralized unit, accessible through a 
network, all of the information available in a company. Data 
Warehousing allows a better distribution of the information 
by no longer presenting the information in the form of raw 
data, but so that it is directly usable by the requestor or client 
(the company's executives, for example). A Data Warehouse 
therefore corresponds to one or more of the company's 
production sites with databases, plus a decision-making base 
that can retrieve the information in a new form (matching 
information that can be represented in different ways, trans- 
forming or deleting information, etc.). At each site, there is 
a process for transforming and routing the data for filling this 
decision-making base. The hardware and software used can 
be different from one site to another. In the prior art, the 
machines at each site require a particular software program 
for interpreting and transforming the information from the 
database or databases of the site, A software program must 
therefore be rewritten for each site. The processes that 
currently exist use the distribution to the various machines 
of a synchronous process known as a Remote Procedure Call 
(RPQ. The various elementary operations (the extraction, 
transfer and loading stages) are mixed in one algorithm. 

SUMMARY OF THE INVENTION 

Hie object of the invention is to restructure the data 
transforming and routing software in order to make it easily 
extendable and to make it open. 

This object is achieved through the fact that the process 
for transforming and routing data between agent servers 
present in some machines and a centra! agent server present 
in another machine is characterized in that an agent server 
comprises autonomous agents that communicate via 
notifications, a storage layer (Storage) and an agent machine 
comprising an execution engine (Engine), a communication 
channel (Channel) and two message queues for the 
notifications, a local queue (mqln) and an external queue 
(mqOut), and in that the execution engine takes a notifica- 
tion from the local queue (mqln), determines and loads the 
corresponding agent for each notification, has the agent react 
to the notification, which agent can then change its status 
and/or send notifications to the communication diannel, 
which stores them in the local queue (Mqln) if they are 
addressed to a local agent and in the external queue (MqOut) 
if they are addressed to an agent of another server. 
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According to one characteristic, the execution engine 
(Engine) of the agent machine executes, after the agent's 
reaction, a save (Save) of the agent's status, a deletion (Pop) 
of the processed notification from the local message queue 
5 (Mqln) and, atomically. a validation (Commit), of the vari- 
ous operations performed that have changed either the status 
of the agent or the status of the message queues. 

According to another characteristic, each message 
includes an optimized matrix clock that can be read by the 
jQ execution engine of the agent machine, thus making it 
possible to process the notifications in their causal sending 
order. 

According to another characteristic, a message is an 
object that contains four pieces of information, i.e., the 
]5 identifier of the sending agent, the identifier of the ctestina- 
tion agent, a clock and the notification. 

According to another characteristic, the agent machine 
controls the distribution to all the entities of the network, the 
reliability of the operations and the scheduling of the 
20 messages, which is established by incrementing a logical 
matrix clock. 

According to another characteristic, the message channel 
performs the following separate transactions: 

a notification sent by an agent is stored locally in a 
25 message queue, i.e., either the local queue (mqln) for 
the notifications addressed to a local agent, or the 
external queue (mqOut) for an agent located in another 
server; 

at the end of the agent's reaction, the two queues are saved 
30 on the disk, together with the reacting agent; 

a function (handleout) of the network extracts each 
notification, in turn, from the external queue (mqOut) 
and sends it to the target agent server; 
a function (handlein) of the network receives the notifi- 
cations from the other servers and inserts them into the 
local queue (mqln) in causal order. 
According to another characteristic, a target agent server 
receives a message, stores it in its persistent storage layer 
(Storage), validates the operation (Commit) and confirms 
the reception of the message to the source server; upon 
reception of this confirmation, the source server deletes the 
message from its external queue (mqOut). 

According to another characteristic, a persistent external 
service ensures the atomic nature of the operations that have 
changed either the status of the agent or the status of the 
message queues (3,4). 
According to another characteristic, the execution engine: 
takes a notification that is in the queue, using the Get 
function, 

50 

sees which agent the notification is addressed to, 
loads the agent corresponding to the notification (Load), 
has the agent react by calling a reaction operation (React); 
the agent can change its status by modifying its image 
55 in memory, and can send notifications by addressing the 
channel that will distribute the notification, in which 
case the notifications are routed to a message queue 
where they are stored in chronological order, 
if the reaction operation is executed, saves (Save) the 
60 status of the agent, which has been changed by the 
reaction it has executed, deletes (Pop) the notification 
that has been processed, and atomically validates all of 
the operations that have been performed (Commit), 
starts again in the program loop in order to process 
65 another notification. 

According to another characteristic, the data transforming 
and routing process is executed between agent servers 
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present in machines of production sites and a central server a network, for example a component Network. In each agent 

present in the machine of a decision -making base, and uses server, an agent machine is defined, which in turn has the 

a particular agent (Event) describing the specific events, in capability to define agents, with a certain agent model. This 

which a status automation is implemented. agent machine manages a certain number of properties. 
According to another characteristic, the status controller 5 which include the atomic aspect of an agent's reaction to a 

comprises various statuses: notification and the delivery of notifications in a causal 

. ^ • L- u . • . sending order. The agent machine comprises various enti- 

a None status m which the event is not active, ^ i c * t-?/^ < *u . u- 

wLiv otoi«^ rruiwu iiiw iiwt refcrcuce to FIG. 1, the agent machine comprises an 

a notification (Schedule) causes the event to pass into the execution engine (1) (Engine), a commimication channel or 

Wait status in which the event waits for its activation message bus (2) (Channel) associated with the execution 

condition to become true; the event delivers itself a engine (1), the channel (2) having two queues, a local queue 

notification that changes it to the Init status; (3) (mqln) and an external queue (4) (MqOut), The agent 

a simple event attains the Run status directly and ail server (10) also includes a storage layer (Storage). The 

executes its action; a composed event signals to its queues (3, 4), the channel (2) and the agents (5) are 

subnevents its agreement for them to start and waits for supported by the storage layer. 

the first of them to attain the Run status in order to pass All the agents (5), the communication channel (2) and the 

into the Run status itself; execution engine (1) of the agent machine are C++ objects 

- - . t . . in a UNIX process. Persistence is obtained by using a 

a simple event then passes into the Done or Fail status ^^^^ ^^^^ 3^ Management System). Most of Ihc 

depending on the status of its action; for a composed objects, and not just the agents, can be stored in the 
event, this change depends on the execution statuses of 20 persistent storage layer. Each object has a unique, persistent 

all the sub-events; identifier. The objects are compatible with a generic inter- 

a event in execution can be killed by sending it a notifl- face in order to be used by common, persistent service 

cation (Eventkill), in which case it passes into the Kill functions. 

status, marking the fact that it has been killed, and then The agents (5) are elementary programming and execu- 
te the Stop status when it has actually been killed; 25 tion entities. The agents (5) are autonomous in order to give 

a terminated event, i.e., in the Done, FaU or Stop status, structure the property of extendibility by making it easy 

can be rc^xecuted by the client by sending it a noti- ^ add and remove agents within the architecture. They are 

fication (Schedule), in which case it is reset and passes "^^^ autonomous by correcUy defining the interface of 

into the Wait status; the event then propagates the ^hese agents. They are also autonomous in terms of com- 
notification to all the sub-events; an event executed 30 mumcaUon. They commumcate using noUficaUons and not 

repeatedly and endowed with a Done status is auto- procedure caUs, which makes it possible to open up the 

maticaUy reset by the agent Scheduler, which sends it architecture. An event is represented by a notification. The 

a notification (Schedule); notifications pass through the channel (2) and are controlled 

„^ , ^ . ^ , by the execution engine (1) (Engine) of the agent machine. 

an event m the Fail or Stop status can be re -executed ' j i- j 7 * - *u 

, . , . *^ ,. 35 The notifications are dehvered to the agents via the com- 

without being reset by sending a notification . ^. , , ^-x i\ . /-v 

/I- . * .^ L J*- i_ «. M.u munication channel (2) (Channel). The agents (5) are per- 

(Eventrestart); when this condition becomes true, the ^ , *u ■ \ * • j * • j u j . .u « • . 

^ , . ^ ' • . . L- u - . sistent objects, their status is deterrmned by data that exists 

event restarts at the pomt at which it was previously * -j *• * a * /b\ - • j 

, *^ 1' / outside any execution structure. An agent (5) is an indepen- 

?' \ .1- I. . • L dent machine that reacts to the notifications. All the agents 

Accordmg to another characteristic, a new function can be . , j . j 1 *u * j . u 

J . I.-. . L J J- .1 J 40 are mdependenl and only the reactmg agents need to be 

added to the architecture by adding an agent class denved 1 j ^ • . 1 « 7» ■ * .u * 

fro th class "Event " & & loaded mlo the memory. A reacting agent is an agent that 

' ^ . .t_ - £ has iust received a notification, and whose behavior is 

Accordmg to another charactenstic. three categories of k„ tv,o «i>oo^*» A.«^t;«« «f tt,- .^»»t ^.^u^a k,, 

"Event" tvoe agents— "SeaEvent " "ParEvent " imposed by the React function of the agem caUed by the 

bvent type agents beqbvent. Parbvent, execution engine (1) (Engine). These reaaing agents can 
"Comp lexEvem ^ype take on other elementary operations ^ ^.^J 

of the "Event type m order to respectively execute them in ^^^^^ ^^^^^^^ ^^.^^ ^ described below. Each 

sequence m paraUel or m accordance with the dependencies ^ ^ .^^^^^^ 

between the various operations. ^^^^^ ^^^^ ^^.^^^ ^ ^^^^^ 

BRIEF DESCRIPTION OF THE DRAWING ^^^^ ^ '^'"^^ '"'^ 

50 is responsible for creating the identifier. The structure of the 

By Other characteristics and advantages of the present identifier must take into account the static location of the 

invention will emerge more clearly through the reading of agents in order to allow the system to keep track of the 

the following description provided in reference to the notifications and to allow local "production" of identifiers 

appended drawings, in which: for the remote agents. The identifier of an agent comprises 

FIG. 1 represents the architecture of an agent server at a 55 three parts, i.e., the identification of the agent server hosting 

site and the algorithm of the execution engine of the agent the creation agent, the identification of the agent server 

machine; hosting the agent created, and a stamp, local to the agent 

FIG. 2 represenls the status controller of an agent ^"^^^ ^''"e. the creation agent. 
"Event"- persistent objects are divided into four classes. All of 

ni^o 1 ^ J .-I ..u 1 UA . 60 the objects having the same properties or the same policy for 

HGS. 3, 4 and 5 respectively represent the class Agent, j u 1 * 1 a j c j 1 

« u 1 L^L 1 « J . « >» aUocatmg identifiers belong to one class. A defined class can 

the class Channel, and the class mcssagcQucue. , j . j c i_ 1 i_ jj- -a 

^ be reused to define a sub-class by addmg specific properties 

DESCRIPTION OF THE DRAWINGS ^ sub-classes inherit properties from the superclass. 

An inheritance graph defines the hierarchy of the classes. 

Schematically, at each production site, there are one or 65 Four basic classes are also defined, the class "globalObject," 

more machines constituted by an agent server. The commu- the class "localObject," the class "notification," and the class 

nications between the agent servers are established through "agent." 
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The objects of the class "globalObjecl" represent the 
configuration data defined by the client at the lime of the 
configuration and shared by the various events. TWo main 
categories are described, the hosts in the class "Host" and 
the derived classes, and the databases in the class "DbCoa- 
fig" and the derived classes. An object of the class "DbCon- 
fig" always points to an object of the class "Host." 

The class "localObject" has been defined so as to provide 
flexibility in the management of persistence. The interface 
for persistence imposes the movements of aU of the objects 
from the storage layer (storage) to the memory and from the 
memory to the storage layer, even if some of the objects 
have not changed during the reaction. Declaring a member 
of an object in this class makes it possible to explicitly 
manage the persistence of this member, iodependendy of the 
persistence of the object itself. 

The objects of the class "notification" represent the mes- 
sages exchanged by the agents. Most of the notifications 
relate to the management of "warehouse" events. 

The agent class represents the "warehouse" events. The 
object class "Agent" defines the generic interface for all the 
agents. The object class "Agent" defines the common behav- 
ior of all the agents. Each agent is an object of one of the 
classes derived from the agent class. The agent class pro- 
vides a mechanism ("swap-in/swap-out") that makes it pos- 
sible to load the agents or to find them in main storage in 
order to activate them and to dump the agents not used since 
a given moment. This is performed by a class 
"AgenlHashTable," which maintains a list of the agents 
already in memory with the date of their last access. This 
class is created using a table (Hashlable) with the agent 
identifier in the first position. The agents can be explicitly 
deleted from the memory by a "Delete" function which 
deletes them from the table (Hashtable). In reference to FIG, 
3, the object class "Agent" (20) defines two functions, the 
"Save" function and the "Load" function, which saves and 
loads the status of the objects into and out of auxiliary 
storage. The agents, identified by their identifiers, are loaded 
automatically by the execution engine (1). The reactive 
behavior is implemented by a "React" function, which 
defines the reaction of an agent when it receives a notifica- 
tion. This function is called by the execution engine (1) of 
the agent machine. 

A first type of "Agenda" or "scheduler" agent (Scheduler) 
manages the chronological execution of "warehouse" que- 
ries. This agent is activated by the reception of a notification 
(Schedule) (32). This agent reacts to this notification by 
selecting the agents it must reactivate by sending an agree- 
ment notification (AgreementScheduler). It manages the 
time and the alarms, and the agents can request to be 
"reactivated" at a certain lime. This agent is present only in 
the central agent server of the decision-making base. 

A second type of agent "factor/* (factory) is present in all 
the agent servers and is responsible for creating objects in 
the server. It makes it possible to manage the creation of 
agents remotely. 

A third and a fourth iypo of agent "UnixEvent" and 
"SigChild" make it possible to manage UNIX processes. 
"Sigehild" exists in a UNIX agent server and captures the 
termination signals of "child" processes. 

A fifth type of agent "Clientproxy" makes it possible to 
transcribe commands ordered by a client into notifications. 
It exists only in the central agent server and supports the 
relationship with the client. 

An agent class "Event" describes the "warehouse" events. 
A "warehouse" query is composed of several elementary 
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tasks called By "warehouse" events. In this agent 
architecture, each event is executed and controlled by a 
separate agent. All the "warehouse" events have a behavior 
an properties in common, which results in a more precise 
definition of a type of agent specific to the Data Warehouse: 
the agent Event. It must be compatible with a certain 
interface and a mode of operation. Like any agent, it is a 
status machine, but in this case the statuses are defined. An 
object "Event" differs from an object "Agent" by the fol- 
lowing attributes: 
an event has a name and a description; 
all the events that form a '*warehouse" query are orga- 
nized hierarchically. The query is identified by an 
identifier of the event at the root of the hierarchy, and 
all of the events retain the identifier of their parents and 
their query; 

the purpose of an Event is to accomplish an action. The 
start of the execution of the action is conditioned by the 
reception of a series of agreement notifications 
(Agreement) (34); 
an object Event uses lists (tables) and produces lists 
reailting from an extraction, transfer, or transformation 
operation, for example. The tables produced are estab- 
lished with the notification of the termination of the 
event's action (ActionDone) (44), and are represented 
by C++ objects in the class localObject; 
an object Event is a status machine; it maintains a status 
variable and reports changes in its status to its parents 
by means of notifications (StatusNotification) (45). 
Two main groups of events can be isolated. A basic class 
(DelegatingEvent) is defined for all the events that perform 
an elementary operation specific to Data Warehousing: 
extraction, transfer, transformation, etc. These various 
operations inherit from the agent Event all the properties of 
this object. A second basic class of service 
(CdmposedEvent) helps to structure the execution of a 
query: a sequential, parallel, or more complex execution. 
This class defines a common behavior for all the composed 
events such as, for example, the management of the list of 
sub-events; create, save, restore, delete. 

The last important event is the UnixEvent, which includes 
the execution of a Unix command. 

Hie status changes possible during the execution of the 
event are represent in FIG. 2. The status graph or status 
controller (30) is implemented in the agent EVENT in C++: 
the None status (3) is the initial status in which the event 
is not active; 

a notification (Schedule) (32) causes the event to pass into 
the Wait status (33) in which the event waits for its 
activation condition to be come tme (True). This con- 
dition depends on a series of agreement notifications 
(Agreement) (34). When this condition becomes true, 
i.e., when all the agreements have been received, the 
event delivers itself a notification (Eventstart) (35) in 
order to start; 

upon reception of this notification (Eventstart) (35), the 

status is set to Init (36); 
a simple event attains the Run status (37) directly and 
executes its action. A composed event signals to its 
sub -events its agreement for them to start, and waits fox 
the first of them to achieve the Run status (37) (it waits 
for the action to have actually started); 
a simple event passes into the Done (38) or Fail (39) 
status, depending on the status of its action, upon 
reception of a notification that the event delivers to 
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itself (respectively 44, 46). For a composed event, this (38) or the Fail status (39), or possibly, if a stop request (30) 
change depends on the execution statuses of all the has been requested, the Stop status (42). The organization 
sub<vents; and synchronization of the elementary tasks are part of a 
an event in execution can be killed by sending it a library. Three particular agent classes, "SeqEvent," 
notification (Eventkill) (40), in which case it passes 5 "ParEvent," and "ComplexEvent," which inherit from the 
into the Kill status (41). marking that it has been killed. class "ComposedEvent " have been created. An object 
and then to the Stop status (42) when it has actually "SeqEvent" executes its sub-events in sequential order. It 
been killed, upon reception of a notification (46); reacts to a notification "EventSurt" by sending its agree- 
a terminated event, i.e., in the Done (38), Fail (39) or Stop mem to the first of its sub-events. When the latter is finished, 
(42) status, can be re-executed by the client by sending 10 it passes to the Done status (38). If one of the sub-events 
it a notification (Schedule) (32). fails, the object "SeqEvent" stops and passes into the Fail 
an even in the Fail (39) or Stop (42) status can be status (39). 
re -executed without being reset by sending a notifica- In the same way, an object "ParEvent" executes its 
tion (Eventrestart) (43). Wten this condition again sub-events in parallel. It reacts to the notification Eventstart 
becomes "true," the event restarts at the point at which is by sending its agreement to all the sub-events. It waits imtil 
it was previously stopped. they are finished and changes its status as a function of their 
In the class "Event," a function (Start) makes it possible ending statuses. An object "Cbmplex Event" makes it pos- 
to change the status and to automatically send the status sible for arbitrary dependencies to be set between sub- 
change to the operation that manages the operation. The events. Since each event or elementary operation, or each 
"Start" function makes it possible, simply by calling it, to 20 "Event," must comply with the status graph, it is possible to 
send the necessary notifications automatically and to change construct an event that will manipulate sub-events of an 
the status. The class has a basic implementation for reactions unknown type, but which are known to comply with this 
10 a certain number of notifications. In particular, there is a graph, so that they are executed sequentially. It is possible to 
procedure in the basic class Event for changing from Wait start two operations in parallel, for example two extractions 
(33) to Init (36). An agent that inherits from Event, i.e., that 25 that do not involve the same databases and the same sites, 
is part of a sub-class of the class "Event," retains this and to re-synchronize at the end of these two operations in 
management. In the standard mode, the status changes to Init order to start another operation, for example a iransforma- 

(36) , then to Run (37), then to Done (38); from Done (38), tion. Adding parallel management, for example, while main- 
the status can return to Wait (33). This is the standard taining the reliability of the agent machine (which is 
schema. The basic implementation is Wait (33), Init (36), 30 described below) is possible as a result of the open and 
Run (37). In more specific classes, it is necessary to control extendable architecture of the process. Thus, different ser- 
the change from Init (36) to Run (37) manually. There is a vices can be added to the machine simply by adding a class, 
basic implementation for reacting to a stop request signal The agent machine ensures the distribution of the 
(40). Upon reception of this Kill notification (41), it will notifications, the transmission of notifications, and the man- 
change to the Kill status (41), then to the Stop status (42). 35 agement of the atomicity of all the operations. The agent 
This implementation can, for example, be loaded into a machine comprises various entities. In reference to FIG. 1, 
sub-class; it is possible in the sub-class to provide another the agent machine comprises an execution engine (1) 
function that will implement this change, possibly separat- (Engine), a communication channel (2) (Channel) associated 
ing it into two stages, from Run (37) to Kill (41), then from with the execution engine (1), the charmel (2) having two 
Kill (41) to Stop (42). There is therefore a basic implemen- 40 queues, a local queue (3) (mqln) and an external queue (4) 
tation for all these status changes. When wishing to define a (MqOut). There are well-defined entities and the program 
particular class, it is necessary to override the class Event (6) of the execution engine (1) uses these various entities, 
and to provide a implementation with a certain number of The interface of the communication channel (2) (Channel) 
functions that follow its rules, and each function, for is provided by the class "Channel." According to FIG. 4, the 
example the Start function, must necessarily call the change 45 class "Channel" (21) defines a function (Seodto), which 
to the Run ^atus (37). In a class, there are variables, each of sends a notification to an identified agent. The object "Chan- 
which contains one of the statuses, and status change nel" is responsible for the target agent. The object "Channel" 
functions, each of which triggers the change, for example provides the following properties: a notification accepted by 
from Wait (33) to Run (37) or from Run (37) to Done (38). the channel (2) (Channel) is sent at some point, and two 
In the class Event, a standard layout is provided, which so notifications with identical source and target agents are sent 
makes it possible to change directly from Init (36) to Run in order of their emission, 

(37) , but a class that is going override it must provide the According to FIG. 5, the class "MessageQueue" (22) 
implementation for the Start function by placing in its code defines an addition function (Push) used by the object 
the change from Init (36), followed by the change to Run "Channel" to add a notification to a queue, and two functions 
(37), Thus, introducing a new agent class amounts to ss (Get and Pop) used by the execution engine (1). 
inheriting, then providing implementation for new The execution engine (1) is provided by the class 
functions, in other words, adding a new function to the "Engine." The class "Engine" is the main part of the 
architecture amounts to adding an agent class derived from architecture. This class "Engine" provides a multiprogram- 
the class Event. ming of the agents. The distributed architecture involves 

The data transfer process is broken down into elementary 60 several agent servers running on different machines. Each 

tasks which pass through the statuses described above; thus, agent is attached to only one agent server, the migrating 

the execution progresses in elementary steps. Each elemen- entities being the notifications. One of the essential proper- 

tary operation must satisfy the status graph. When, for ties of the execution machine in an agent server is the 

example, a service is requested of an elementary operation, transactional execution of an agent reacting to a notification: 

the operation will pass into Wait (33) and Init (36), and the 65 either all of the actions, i.e., the notifications sent and the 

agent machine (which is described below) guarantees that status changes of the agents, are executed, or none of the 

the operation attains, at a given moment, the Done status operations are executed. Each object of the class "Engine" 
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is associated with ao object of the class "MessageQueue" them (15) ioto the local queue (3) (mqln) in causal 
de&oed when the object "Engine" is created. The execution order. When the target agent server receives the 
engine (1) (Engine) provides a program loop (6) which message, it stores it in its persistent storage layer 
successively takes the notifications from the local message (Storage), validates the operation (Commit) and con- 
queue (3), and calls and executes the appropriate reaction 5 firms the reception of the message to the source server 
fimction (React) of the target agent. Upon reception of this confirmation, the source server 
In reference to the algorithm (6), of the execution engine deletes the message from its external queue (4) 
(1) (Engine) represented in FIG. 1, (MqOut). The message is then guaranteed to have been 
the execution engine (1) takes a notification that is in the stored persistently by the agent server 

local queue (4) using the Get function (7); The operations for saving the agent's status, for deleting 

the execution engine (1) sees which agent the notification the processed notification (Save, Pop) and for validating the 

is addressed to; sent notifications must be intimately linked. The atomicity 

the execution en^e (1) loads the agent that corresponds results from these three operations. It is essential, for 

to the iKJtification (Load)* example, not to save the agent and leave the notification in 

the execution engine (1) ha^ the agent react by caUing a the local queue (3), so that the notification is not processed 

reaction operation (React) (8). Tlie agent can change its ^ ^^"^ ^ ^ ^""^'^ operaUons. Uns part 

status, i.e., modify its image in memory, and can send ^ °?^°^g*^ atomicaUy. In order to ensure atomicity 

(9) notifications, i.e.. address the channel (2), which ^'^^^^^^ these two operations, the process relies on a per- 

will distribute (U) the notification. The notifications f "^^'^^ ^^'^^^ 

are then routed to a message queue (3, 4) or they are ^6^°^ ^^t local queue and the external queue simulta- 
stored in chronological ordtr; "^20 neously. Everythmg is kept in the same persistent service 

.r .f,. that provides this atomic Operation. When changes are made, 

if everythmg runs correctly, I.e.. if the reaction operation .1. • . . • u • r.i. m J 

, t 1 . 1 Vl * . the persistent service saves the old version of the files, and 

is executed completely, the execution engmc executes u i-j .- .* « • n j • .l* 

\ r .i_ *» * L • L I. i_ when the vahdation operation commit is called m this 

a save (Save) of the agent s status, which has been ... ■ . • A. • . . • .i_ . .t_ 

. J u *L •* L .J J I persistent service. It IS the persistent service that ensures that 

changed by the reaction it has executed, a deletion ^ f. . . ... - ^^ ^ . • • 1 j 

(pop) of the processed notification from the local " *e oj^aUon is executed atomically TTie atoinicUy uicliides 

^ '^'^ '^>.-x J 1-j r II *t. the modifications m the agent, but also the messages it has 

message queue (3), and a vahdation of aQ the opera- . . .1. . • j j • 1. j -f^ ... 

IL . L L _r J /r^ '.\ All r fi- scut to thc outsidc and the message it has used. Three thmgs 

Uons hat have been perforined (Commit) AU of these ^re executed atomicaUy. 

Operations have changed either the stanis of thc . . i • *u .u u • *t. 

* *, c.L /I >i\ J If a problem occurs during the reaction, the changes m the 

memory, or the status of the message queues (3, 4) and j . . n j nu 1 \ * * 
* • 1 1 u-u - J c j ui 30 database are cancelled (Rollback). Dependmg on the type of 

the matrix clock, which IS defined below; u * r jj-.- 1 :* 

. error, It may be necessary to perform additional operations 

then the execution engine, (1) starts agam in the loop m i„orfer to re^ynchronize the database and the objects of the 

order to process another notification ^^ ^^^^ continue. T^o types of 

Ttis process is distnbuted among aU of the agent servers ^^^^^ ^„ ^^^^^ ^ j^,^,^ 

of the machines. The component Network (12) is respon- 3, i, .-exception" in 

sible for establishing commumcation between the agent ^ . . .«^„„„ r^^^Zt,.^^\ » 

. * C++. The serious errors (second type) result m a signal that 

servers. When the agent server is initialized, the potential .u.^ » .,«-t 1- ik« n^. «L„ .u. o„™-.:~„ k. 

,, ..... . makes them apparent. In the first case, the exception can be 

agents created beforehand are aU on the d^ m persistent 1,^^;^^ ^, ^ ^^^j, 

space. Wien the execution engine (1) sees that it must make distinguished by the source of the exception. If the exception 
a parucular agent react because there has been a notification ^ ^as been raised during the agent's reaction, the agent is 

to this agent, it will store it in the memory of the proces, ^^^^ ^ j^jj^ operations in the database are 

possibly from the image saved on the disk. This is the ^^^^u^a tuJ^^^^ v tu« 

f:. „/ ^ - . , . c ance Ilea and the message queue is re-stored, rne execution 

Load operation, a movement from the disk to the memory. rt\ .i,^ if . »u-> 

_ ^ , ' , . . , ^ engme fl) can process the next notincation. It not, the 

Part 01 the source acent s reaction is to deuver new * *j* i* u'l. 

. ' . a^uivv a^viii ivai^iiv^Li 1^ "''7 exception either represents an expected signal, in which case 

notifications to the other, so-called target agents. The send- .u^ Jl.^^ a- • . ^ -i . % • -j 

- , . ' . n » . ^ . ^5 the corresponding notification is created and sent, or it is id 

mg of the notifications is part of the transaction of the the occurrence of a serious error In this case, the processing 

reactmg source agent server. The possible use of the noti- ^ ^^^^ j ^ .^^^ ^ 1^ 

fication IS part of the transaction of the reacnng target agent ^^^^ ^. ^^^^ 

server The transmission of the noufication from the source ^^^^^^^ ^ notification, then it is processed in the 

agent server to the target agent server is not part of the ^^^^ ^ ^^^^^ ^ ^^^^^^ 

agent s reactiori but is achieved via the chamiel (2), which ^^^^ ^^^^^ notification or from the 

executes the foUowing separate transacuons: ^^^^^^ ^^J^ ^^^^ notification is ignored, 

a notification sent by an agent through the channel (2) is ybe algorithm (6) of the agent machine manages the 

stored locaUy in a message queue, i.e.. either the local distribution of the notifications to aU the entities in the 
queue (3) (Mqln) for the notifications addressed to a 55 network, and the reliability of these operations, especiaUy 

local agent, or the external queue (4) (mqOut) for an i^eir scheduling. It ensures that the messages have actuaUy 

agent located m another server; ^^^^^ deUvered to the destination agents in the order in which 

at the end of the agent's reaction, the two queues (3, 4) are they were sent. The relationship between the notifications is 

saved on the disk, together with the reacting agent. The established simply by numbering the messages, by incre- 

changes in the agent and in the message queues (3. 4) naenting a logical clock. A logical clock makes it possible to 

take place in the same transaction, and arc eslabli^ed obtain a certain number of properties (the causality property 

at thc same time; of the channel, for example), 

a function "handleout" of the network Network extracts a matrix clock is included in each message, and each time 

(15) each notification, in mra, from the external queue a message is received, a part of the algorithm will first read 
(4) (MqOut) and sends it (13) to the target agent server; 65 the position of the clock, 

a function "handle in" of the network Network receives A message is an object that contains four pieces of 

(14) the notifications from the other servers and inserts information: the identifier of the sending agent, the identifier 
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of the destinaiioD agent, a cbck, and the notification. If there 
are n sites, the clodc is composed of a matrix oxn, since the 
status of the communication of each site with eadi other site 
must be maiotaioed. Each machine will try to maintain its 
own view of the global communications, even the commu- 5 
nicalions between two other machines. A matrix nxn corre- 
sponds to a very high-volume piece of information. An 
optimization of the transferred data is performed. A part of 
the sending algorithm calculates the part of the clock actu- 
ally used and transfers only this part. lO 

Thus, the invention provides the agent machine with the 
property of failure resistance, since a large process is divided 
into small elementary processes. In particular, it applies to 
Data Warehousing and to the distribution of the process. The 
data transforming and routing process is broken down into 15 
elementary steps. In a distributed process, several machines 
are impacted and the risk of a machine or network failure is 
high. Breaking the process down into steps makes it possible 
to isolate one piece in particular, i.e., the transmission of the 
notification from one machine to another. 20 

The agent machine is distributed. The agent machines 
commimicate with one another; the agents do not see this 
distribution since they communicate via notifications and 
address their local agent machine. An agent's reaction 
remains local. The atomic reaction, which is to change the 25 
status and to send other notifications, remains local. Thus, 
none of the network failures have any influence on this 
reaction. Moreover, the transmission of the notifications 
itself is an atomic operation. Each notification sent is an 
atomic operation, which makes it possible to handle network 30 
failures by saving the notifications while waiting for the 
machine or the network to return to normal operation. 
Transmission errors, site failures and network failiu'es are 
handled at the agent machine level. 

A notification is something asynchronous. When a noti- 35 
fication is sent, the program can continue in an asynchronous 
mode. When the notification is sent, it is certain that it will 
be received and that the consequence of the message that has 
been sent will be executed, and there is no need to wait for 
the exact result. This corresponds to the property of atom- 40 
icity in the delivery of the messages and in the order of 
delivery of the messages. With the rescheduling of the 
messages in causal order, and with the reliability of the 
delivery of the messages, the sender can completely rely on 
the properties provided, so as not to have to re-synchronize 45 
in order to be sure that the operations will be able to be 
executed in the correct order. Thus, the handling of data 
errors is eliminated from the program. 

Guaranteeing this reliability of the network, and this 
scheduling at the agent machine level, makes it possible to 50 
extract a relatively simple status graph, which is easy to 
handle since includes only the applicable errors. The event 
level (Agent Event) provides something of a structure for the 
various statuses passed through, and makes it possible, in 
particular, to handle the errors that are currently applicable 55 
at this level, which can occur at each step. The applicable 
errors are handled by the status automation. By breaking 
things down into elementary operations (transfer step, load- 
ing step), and by having a controller that reacts correctly, for 
which there is a certainty of its reacting correctly, it is 60 
possible to identify the step and to restart this step only. 

The agent Event is an agent specific to Data Warehousing. 
The agent machine according to the invention can therefore 
be used in the Data Warehousing field by manipulating this 
specific agent as defined in the invention. Of course, the 65 
agent machine can be used in other fields, using agents 
specific to each of these fields. For example, the agent 
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machine can be used in the firewall field (Netwall). A 
firewall authorizes or inhibits connections to a network, and 
all the information related to the actions it executes are 
placed in a log, a file (Log) whose size increases rapidly. The 
process according to the invention can be rewritten in 
another language, and can be used to manage this file (Log) 
using agents specific to the firewall field (NeiwallEvent). 
These specific agents use the same model for changing the 
status of the agent "Event** specific to Data Warehousing, but 
are adapted for managing data of the file type specific to the 
apphcation of the fire wall. 

While this invention has been described in conjunction 
with specific embodiments thereof, it is evident that many 
alternatives, modifications and variations will be apparent to 
those skilled in the art. Accordingly, the preferred embodi- 
ments of the invention as set forth herein, are intended to be 
illustrative, not limiting. Various changes may be made 
without departing from the spirit and scope of the invention 
as set forth herein and defined in the claims. 
What is claimed is: 

1. A process for transforming and routing data between 
agent servers in one or more of a phirality of first machines 
and a central agent server of a second machine, wherein an 
agent server comprises autonomous agents that communi- 
cate via messages, each message having at least a 
notification, and wherein each agent belongs to one type of 
agent and is designated by an agent identifier, the agents 
being shared out into at least two types, with each type of 
agent handling only one type of notification, the agent server 
also comprising a storage layer and an agent machine having 
an execution engine, a commimication channel and two 
message queues associated with the notifications, namely, a 
local queue and an external queue, the process comprising 
taking a notification from the local queue, by the execution 
engine, determining and loading a corresponding agent by 
its agent identifier for each notification, depending on which 
type of agent is required to handle the notification, causing 
the agent to react to the notification, such that the agent can 
then change its status and send addressed notifications to the 
communication channel, storing the notifications in the local 
queue if said notifications are addressed to a local agent and 
in the external queue if said notifications are addressed to an 
agent of another server. 

2. A process for transforming and routing data according 
to claim 1, characterized in that the execution engine of the 
agent machine executes, after the agent's reaction, a save of 
the agent's status, a deletion of the processed notification 
from the local message queue, and atomically, a validation 
of various operations performed that have changed either the 
status of the agent or the status of the message queues. 

3. A process for transforming and routing data according 
to claim 1, characterized in that each message includes an 
optimized matrix clock that can be read by the execution 
engine of the agent machine, thus making it possible to 
process the notifications in their causal sending order. 

4. A process for transforming and routing data according 
to claim 2, characterized in that each message includes an 
optimized matrix clock that can be read by the execution 
engine of the agent machine, thus making it possible to 
process the notifications in their causal sending order. 

5. A process for transforming and routing data according 
to claim 1, charaaerized in that a message is an object that 
contains four pieces of information, including an identifier 
of the sending agent, an identifier of the destination agent, a 
clock and the notification. 

6. A process for transforming and routing data according 
to claim 1, characterized in that the agent machine manages 
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distribution to all entities of the network, reliability of 
operations, and scheduling of messages which is established 
by incrementing a logical matrix clock. 

7. A process for transforming and routing data according 
to claim 1, characterized in that the communication channel 
performs the following separate transactions: 

storing a notification sent by an agent locally, cither in the 
local queue for notifications addressed to a local agent, 
or in the external queue for an agent located in another 
server, 

at the end of the agent's reaction, saving the two queues 
on a disk, together with the reacting agent; 

extracting by a first function of the network each 
notification, in turn, &om the external queue and send- 
ing the extracted notification to a targeted agent server; 

receiving the notifications from the other servers by a 
second function of the network and inserting the noti- 
fications into the local queue in causal order. 

8. A process for transforming and routing data according 
to claim 1, characterized in that when a targeted agent server 
receives a message, the targeted agent server stores the 
message in a persistent storage layer, validates the operation 
and confirms the reception of the message to a source server; 
and upon reception of this confirmation, the source server 
deletes the message from the external queue of the source 
server. 

9. A process for transforming and routing data according 
to claim 1, characterized in that a persistent external service 
ensures the atomicity of the operations that have changed 
either the status of the agent or the status of the message 
queues. 

10. A process for transforming and routing data according 
to claim 1, characterized in that the execution engine 
includes a program that: 

takes a notification that is in the local queue, using a Get 
function; 

identifies which agent the notification is addressed to; 
loads the identified agent corresponding to the notifica- 
tion; 

causes the agent to react by calling a reaction operation 
such that the agent can change its status by modifying 
its image in memory, and can send notifications by 
addressing the channel that will distribute the 
notification, in which case the notifications are routed 
to a message queue where they are stored in chrono- 
logical order; 

if the reaction operation is executed, saves the status of 
the agent, which has been changed by the reaction it has 
executed, deletes the notification that has been 
processed, and validates all of the operations that have 
been performed; and 

starts again in the program loop in order to process 
another notification. 

U. A process for transforming and routing data according 
to claim 1, charaaerized in that the process is executed 
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between agent servers present in machines of production 
sites and a central agent server present in a machine of a 
decision-making base, and uses a particular agent that 
describes specific events, in which a status controller is 
implemented. 

12. A process for transforming and routing data according 
to claim 11, characterized in that the status controller com- 
prises various statuses including: 

a None status in which the event is not active; 

a notification status that causes the event to pass into a 
Wait status in whidi the event waits for its activation 
condition to become '*true"; at which time the event 
delivers itself a notification that changes the event to an 
Init status; 

a first simple event status that attains a Run status directly 
and executes its action; 

a first composed event status that indicates to sub-events 
agreement for the sub-event to start and waits for the 
first of the sub-events to attain a Run status in order for 
the composed events to pass into the Run status; 

a second simple event which then passes into a Done or 
Fail status, depending on the status of the simple event 
action; 

a second composed event, which passes in the Done status 
on the execution statuses of all the sub-events; 

an event in execution status that can be killed by sending 
to the event in execution status a notification, in which 
case the event passes into a Kill status, marking the fact 
that the event has been killed, and then to a Stop status 
when it has actually been killed; 

a terminated event, i.e., in the Done, Fail or Stop status 
that can be re-executed by a client by sending the event 
a notification, in which case the event is reset and 
passes into the Wait status; the event then propagates 
the notification to all the sub-cvenls; an event executed 
repeatedly and endowed with a Done status is auto- 
matically reset by an agent Scheduler, which sends the 
event a notification; and 

an event in the Fail or Stop status can be re-executed 
without being reset by sending a notification, and when 
this condition becomes "true," the event restarts at the 
point at which it was previously stopped. 

13. A process for transforming and routing data according 
to claim 12, further comprising adding a new function by 
adding an agent class derived from the class "Event." 

14. A process for transforming and routing data according 
to claim 11, characterized in that three categories of "Event" 
type agents — "SeqEvent," "ParEvent" and 
"complexEvent" — take on elementary operations of the 
"Event" type in order to respectively execute them in 
sequence, in parallel, or in accordance with the dependen- 
cies between the various operations. 
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